:W&Bá¾R×lÅM g49l8az9ª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0V?Tjù‚" ngn0lgv5jtal g49l8az9ŠÕ g49l8az9juju11-uc-san-diegoSPAtop"™ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.22144442499851164 epochs‚10  hidden_dim‚64 ' learning_rate‚0.0008166113137007488  num_attention_heads‚8  num_branch_encoder_layers‚7  num_embedding_layers‚10  num_encoder_layers‚6Bjjtttt_blocks_g49l8az9bs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pŠ ÷’ëÔàÙígš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64X%ÑÞ7 g49l8az9&  _runtime‚0  _wandb.runtime‚0îüRjÒä -Linux-6.8.0-138-generic-x86_64-with-glibc2.36CPython 3.12.10 ö’ëÔ°Øë¯*-of*=/data/spatop/options_files/sweep_tttt_jj/config_g49l8az9.json*-l* /scratch/logs*-n*jjtttt_blocks_g49l8az92-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pr/opt/conda/bin/pythonº ottoc6qt6a6yixhe6mbax6fslbubc0t3¢ö”áÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ ¥¶ã$ª€ äÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-8cb16037-0025-0eda-9425-82f6843a99c3Ê13.2º ottoc6qt6a6yixhe6mbax6fslbubc0t3µª½Ç2  requirements.txtIa]4W g49l8az9ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64S&«Y g49l8az9ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64”ÿ? *  g49l8az9b trainer/global_step2Á|؃\  g49l8az9ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64;`°¶/  g49l8az9b*"trainer/global_step2Á®ÎK  g49l8az9j: ÷’ëÔÐç±Ì*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] •>ëÿ’  g49l8az9j€  ù’ëÔÐëÂAî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 498 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 13.6 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ =b«[ g49l8az9jÉ ù’ëÔÐî¾B·Trainable params: 14.3 M Non-trainable params: 86 Total params: 14.3 M Total estimated model params size (MB): 57.161 Modules in train mode: 5928 Modules in eval mode: 0 Total FLOPs: 0 øa4ºp g49l8az9j_ ù’ëÔ ³×BPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). à´ýòz g49l8az9ji ù’ëÔøÔÞBZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update).  !x g49l8az9jg ù’ëÔÐöåBXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). PùÐ; g49l8az9jn ù’ëÔ؈íB_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). & hy g49l8az9jh ù’ëÔ Ð÷BYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). œT µs g49l8az9jb ù’ëÔ€–€CSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). éók"p g49l8az9j_ ù’ëÔ˜ù†CPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ‹ÿ—Es g49l8az9jb ù’ëÔ ŽCSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). )Cñm g49l8az9j\ ù’ëÔ€Ú“CMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). þ+çbl g49l8az9j[ ù’ëÔèÌšCLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). Œc:Ù g49l8az9*Ç  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"d ‹ê g49l8az9jØ ù’ëÔÀ¹ôIÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ØÃ=M g49l8az9j» ù’ëÔˆøJ«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. …À'òë g49l8az9jÙ ü’ëÔÐïÞÊÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. /òÓP g49l8az9j¾ ü’ëÔȃõÊ­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. =neƒ‚(:“ †“ëÔן֗ gpu.0.gpu‚65.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.895660400390625* gpu.0.memoryAllocatedBytes‚ 15372255232 gpu.0.temp‚34.0 gpu.0.powerWatts‚129.043' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.808600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]¦¬”J‚(:À †“ëÔúö°¢% disk./.usageGB‚1176.8559341430664 disk.sda3.in‚0# memory_percent‚99.9706506729126" proc.memory.rssMB‚ 2933.1796875 cpu‚25.661454557091744 proc.cpu.threads‚17 disk.drbd1260.in‚0) disk./.usagePercent‚78.97978918824066 disk.nvme0c0n1.in‚ 0.0703125$ disk.nvme0c0n1.out‚ 45.2978515625 disk.dm-0.in‚ 0.0703125 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 0.44140625 network.sent‚38706 disk.dm-1.out‚ 0.44140625 disk.sda1.out‚0 disk.drbd1260.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.out‚ 45.2978515625 disk.sda3.out‚ 0.44140625 network.recv‚38158 disk.dm-0.out‚ 45.2978515625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.0703125& proc.memory.availableMB‚ 4.80859375* proc.memory.percent‚17.902708053588867,?(P’ ‚(:ˆ •“ëÔ㯕 gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.895660400390625* gpu.0.memoryAllocatedBytes‚ 15372255232 gpu.0.temp‚33.0 gpu.0.powerWatts‚137.077' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4154 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å“í9W!‚(:Í •“ëÔÊŸ´˜ disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 0.08203125! disk.dm-0.out‚229.94287109375 disk.dm-1.in‚0 disk.sda1.in‚0$ memory_percent‚99.96182918548584 network.sent‚58266% disk./.usageGB‚1176.8321723937988 disk.drbd1260.out‚ 0.00390625 disk.sda2.out‚0$ disk.nvme0n1.out‚229.94287109375 disk.sda3.out‚ 0.9921875 disk.dm-0.in‚ 0.08203125 disk.sda1.out‚0 disk.nvme0n1.in‚ 0.08203125 disk.dm-1.out‚ 0.9921875) proc.memory.percent‚17.90332794189453 cpu‚25.534604436096412 proc.cpu.threads‚17 network.recv‚71344 disk.drbd1260.in‚0 disk.sda.out‚ 0.9921875 disk.sda2.in‚0& disk.nvme0c0n1.out‚229.94287109375& proc.memory.availableMB‚ 6.25390625 proc.memory.rssMB‚ 2933.28125) disk./.usagePercent‚78.97819451730987n–­"‚(:… ¤“ëÔÂñù– gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.489' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8978 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”UM‰Z#‚(:Ð ¤“ëÔ™âÁš disk.sda3.in‚0! disk.dm-0.out‚732.60693359375 disk.dm-1.in‚0 disk.dm-1.out‚ 1.6796875( proc.memory.availableMB‚ 2364.1328125 proc.memory.rssMB‚ 2933.4375 network.sent‚63478 disk.sda2.in‚0 disk.nvme0n1.in‚ 178.5234375$ disk.nvme0n1.out‚732.60693359375! disk.nvme0c0n1.in‚ 178.5234375 disk.drbd1260.in‚0 cpu‚25.45595608405244 proc.cpu.threads‚17$ disk./.usageGB‚1176.855712890625 disk.dm-0.in‚ 178.5234375 disk.sda1.out‚0* proc.memory.percent‚17.904281616210938 network.recv‚75255) disk./.usagePercent‚78.97977433980368 disk.sda2.out‚0 disk.sda3.out‚ 1.6796875& disk.nvme0c0n1.out‚732.60693359375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.00390625$ memory_percent‚85.57047843933105 disk.sda.in‚0 disk.sda.out‚ 1.6796875 ô¯Š$‚(:† ³“ëÔŸà±– gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~{Ô`%‚(:Ö ³“ëÔÓÅ“™ disk.sda2.out‚0) proc.memory.percent‚17.90447235107422& disk.nvme0c0n1.out‚916.52783203125 disk.dm-1.out‚ 2.07421875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.01171875 disk.sda.in‚0 disk.sda.out‚ 2.07421875 disk.sda3.out‚ 2.07421875 proc.cpu.threads‚17 network.sent‚85586 disk.dm-0.in‚ 178.5234375 disk.nvme0n1.in‚ 178.5234375$ disk.nvme0n1.out‚916.52783203125 disk.sda3.in‚0) proc.memory.availableMB‚ 2361.58984375) disk./.usagePercent‚78.97977561984135! disk.dm-0.out‚916.52783203125 disk.sda2.in‚0$ memory_percent‚85.58599948883057 proc.memory.rssMB‚ 2933.46875 cpu‚25.40372920178731 network.recv‚106023% disk./.usageGB‚1176.8557319641113! disk.nvme0c0n1.in‚ 178.5234375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0V“3&b/ lr-AdamW/pg1"trainer/global_step2:JP;¼$õ3'b/ lr-AdamW/pg2"trainer/global_step2:JPör -() _wandbruntime‚66  _runtime‚66ÝièÀ)» & lr-AdamW/pg1‚3.309673645271852e-05 & lr-AdamW/pg2‚3.309673645271852e-05 trainer/global_step‚49 % _timestamp‚1.7885290819122531e+09 _runtime‚ 66.066247851 _step‚0O&sÂ*½ & lr-AdamW/pg1‚3.309673645271852e-05 & lr-AdamW/pg2‚3.309673645271852e-05 trainer/global_step‚49 % _timestamp‚1.7885290819122531e+09 _runtime‚ 66.066247851 _step‚03;¼p+bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP”&bAn,bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP#x ÷k-bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP¤²x•m.bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPÜwO™b/b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPªd”f0bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP¢ïÕc1b_ á=Ân3bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP帨ñi4be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP;FNïp5bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPbó †b6b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPÉ“ æn7bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP„ÐCîr8bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP8\ãf9bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP)_9ƒh:bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP]ª{bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPCí%¥l?bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPpš‚ll@bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPËûa¾qAbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPÓ[±9cBb_ DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP;éédjZbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP)a(©n[bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPDL»n\bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP::_äj]bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP¦ÿîPn^bj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP?‹¥Ìo_bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPšBl`bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPÀ\-mabi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPÁwI&tbbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPÅÆcYecba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP]{ DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPtr$vqbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPK,€|hrbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPmÛù_sb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP¶Zx‰jtbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPÜ@æ8nubj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPp/'gvbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPdÈñœiwbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPÝžÍfxbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPU[!ölybh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPYê\Tazb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP<£S\f{bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPsÂîåg|bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP€ü~q}bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPí)BÑi~be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP*!$âfbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP’]pµe€b` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPji$èsbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPýªÖÅq‚bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPèoÁ}oƒbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPÇ_5\j„be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JPuúín…bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP¾ê‚þn†bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPú­sBt‡bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP¾õû™^ˆbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPA~E f‰ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPãu|ŽhŠbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPË’—p‹bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP¶µ>mŒbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP{²%albg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPiû9…bŽb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP4¢|Wfba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP5Á÷Anbi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP¢}xØs‘bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPü‰øt’bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPÏ+o“bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP~»i¾l”bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPKI@Zl•bg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP¦˜Â$r–bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPôü´{k—bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPfÇÜãk˜bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPhzüít™bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPÜ¢s„_šbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPO¿!@h›bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP%yšmœbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP]obj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPîOŠdtžbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPï¹ÿº`Ÿb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP(Î+&i bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPƒh¡bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPûÜîp¢bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP0r g£bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP†°[ l¤bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPQÍsj¥be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPüS.¦) _wandbruntime‚66  _runtime‚66g駀O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1284 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1351346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚926241 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚926241 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1890106813040 % _timestamp‚1.7885290819142044e+09 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2079430270464 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2200444053504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2101297963520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated]± Ì‚582 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1986974270336 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2252739 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚926239 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2277587 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚926239 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051901 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966132736 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚548820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30506 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _step‚1 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1982035655680 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1987220283232 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2079179365888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1982053658624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564257792 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚899466 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚ 66.068145289 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2200746797056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1890123852400 trainer/global_step‚49 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966132736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚349951 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 297953792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 97143710208 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚548818 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 5725224964}ö]Š'¨„O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1284 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1351346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚926241 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚926241 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1890106813040 % _timestamp‚1.7885290819142044e+09 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2079430270464 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2200444053504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2101297963520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1986974270336 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2252739 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚926239 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2277587 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚926239 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051901 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966132736 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚548820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30506 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _step‚1 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1982035655680 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1987220283232 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2079179365888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1982053658624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564257792 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚899466 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚ 66.068145289 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2200746797056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1890123852400 trainer/global_step‚49 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966132736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚349951 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 297953792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 97143710208 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚548818 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \Y_©bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JP?Üñqªbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP®­Óa«b\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JP(–¨hf¬ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP2=0o­bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPñ¿ Ad®b_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPÀç9§mÂbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPÒT‰ÛpÃbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPj]¶™hÄbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPãd‰ÓoÅbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPddBeÆb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JP†S‡‚eÇb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPP$ÐqnÈbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPC•0dÉb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPNÝqŸpÕbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPbüîEmÖbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP ¨òq×bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP \Ÿ¬sØbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP'Ô öiÙbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP0›™rÚbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPßgÛbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP À±DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPÈfŸÌkçbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPí!Fœfèba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPÆúnhébc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP3SÏDmêbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP‘ÜÙm\ëbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPNÙ phìbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPê­ Fmíbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP׃ùjcîb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JP!©–¨jïbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP͸Óqðbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP3µ—¶kñbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPËÇ:!]òbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPÌ¥Uhóbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP.ŽÒí_ôbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP5?RÅkõbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPªˆüiöbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPsqše÷b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPÔRˆiøbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPPi›™iùbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP2c²eúb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPX+Qþlûbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP/cwübr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP\ÔF~^ýbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JP^*æ5`þb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JP]´@9lÿbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPè‚­Äo€bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP‹|±Ôbb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPªÓf‚ba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPê…—Ëiƒbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPXzkbl„bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP?Á­œ`…b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPb©ÛÏg†bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPa±¢Ûj‡be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPåïU nˆbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPMÙW‚p‰bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPšàøxfŠba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPËÝc¢e‹b` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPòH¢hŒbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JP·Ý.ðobj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPÑð-­mŽbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPâú¬Ôsbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP°Ëû·lbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JP  ;fn‘bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP'Ûíj’be BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPC cm“bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JP,Rr”bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPZ…öm•bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JP=£žr–bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPj$›n—bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPDj„Om˜bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPÃÕGq™bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPÕÙ“^šbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP-´R¬e›b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP8å ­dœb_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP¥ÐÖržbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÖá$VŸbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP^ür bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP{ŽªÖa¡b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP‡Êe¢b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPÑ‘EJj£be BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPô™).¤) _wandbruntime‚67  _runtime‚67 { ¥‹M OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2021339695616 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2026463116746 _step‚2 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 99107736064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2021321692672 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2322580 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚356998 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚559622 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 325796352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2120680346624 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚944514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1284 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚944516 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚944514 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2244120665088 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 101421715968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788529083303753e+09 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2297706 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1353192 MDDeviceStatsMonitor.on_train_batc°úœ’h_end/inactive_split.small_pool.peak‚1265 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚917317 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2021339695616 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚559620 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2142981933568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 98825946282 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 99340651008 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2021321692672 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚916618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 297953792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 99054920026 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚49 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 99340651008 _runtime‚ 67.459012928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1353192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1927637170464 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚944516 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚357695 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 101138731520ÄG¬e•&¦M UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2120680346624 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚944514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1284 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚944516 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚944514 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2244120665088 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 101421715968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788529083303753e+09 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2297706 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1353192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1265 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚917317 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2021339695616 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚559620 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2142981933568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 98825946282 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 99340651008 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2021321692672 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚916618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 297953792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 99054920026 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚49 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 99340651008 _runtime‚ 67.459012928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1353192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1927637170464 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚944516 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚357695 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 101138731520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2021339695616 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2026463116746 _step‚2 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 99107736064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2021321692672 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2322580 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚356998 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚559622 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 325796352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0*mÚ@§b; loss/FRt4/detection_loss"trainer/global_step2:JP;ÕS@¨b; loss/FBt1/detection_loss"trainer/global_step2:JP™Ö§C©b> loss/SRqqt3/assignment_loss"trainer/global_step2:JPþQk˜Bªb= loss/SRqqt3/detection_loss"trainer/global_step2:JPo‚pC«b> loss/SRqqt4/assignment_loss"trainer/global_step2:JP–!ÚÙ@¬b; loss/FRt3/detection_loss"trainer/global_step2:JP¶k'ë7­b2 loss/total_loss"trainer/global_step2:JPy˜´A®b< loss/FRt3/assignment_loss"trainer/global_step2:JPÂØA¯b< loss/FRt4/assignment_loss"trainer/global_step2:JPAW,@°b; loss/FRt2/detection_loss"trainer/global_step2:JPòB±b= loss/SRqqt2/detection_loss"trainer/global_step2:JPӕŘ@²b; loss/FBt2/detection_loss"trainer/global_step2:JPçD]@³b; loss/FBt4/detection_loss"trainer/global_step2:JP«°RÉA´b< loss/FRt2/assignment_loss"trainer/global_step2:JPZ[,èAµb< loss/FBt3/assignment_loss"trainer/global_step2:JPñ!Ž@¶b; loss/FBt3/detection_loss"trainer/global_step2:JPdGBA·b< loss/FBt4/assignment_loss"trainer/global_step2:JPÇjÆŽ@¸b; loss/FRt1/detection_loss"trainer/global_step2:JPßû}dB¹b= loss/SRqqt1/detection_loss"trainer/global_step2:JP¾ÛÙ‚Aºb< loss/FBt1/assignment_loss"trainer/global_step2:JPyÚÎZA»b< loss/FBt2/assignment_loss"trainer/global_step2:JPþqUB¼b= loss/SRqqt4/detection_loss"trainer/global_step2:JP– ¿F-½b( epoch"trainer/global_step2:JP¼¸äùA¾b< loss/FRt1/assignment_loss"trainer/global_step2:JPÒ€}^C¿b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP¤Ð+CÀb> loss/SRqqt2/assignment_loss"trainer/global_step2:JPïÑùý.Á) _wandbruntime‚67  _runtime‚67´ÁBFÂÀ _step‚3 epoch‚0 /loss/FRt1/assignment_loss‚7.348953074879116 1loss/SRqqt1/assignment_loss‚5.410686583132358 1loss/SRqqt2/assignment_loss‚5.447135302485252 .loss/FRt4/detection_loss‚6.180651525301593 .loss/FBt1/detection_loss‚3.801165404475548 1loss/SRqqt3/assignment_loss‚5.416716808226051 0loss/SRqqt3/detection_loss‚4.130848969991614 % _timestamp‚1.7885290833077836e+09 0loss/SRqqt4/assignment_loss‚5.44988735516866 .loss/FRt3/detection_loss‚4.234842791088989 %loss/total_loss‚116.8059185939568 /loss/FRt3/assignment_loss‚7.250747737430391 /loss/FRt4/assignment_loss‚7.376896398408072 .loss/FRt2/detection_loss‚4.092388374606768 1loss/SRqqt2/detection_loss‚3.2992478782425123 .loss/FBt2/detection_loss‚4.498782691028383 _runtime‚ 67.462099078 /loss/FBt4/detection_loss‚7.1976646346890405 /loss/FRt2/assignment_loss‚7.398452451494005 0loss/FBt3/assignment_loss‚2.7419240053962257 .loss/FBt3/detection_loss‚5.263868105762145 0loss/FBt4/assignment_loss‚2.7755148825438125 trainer/global_step‚49 .loss/FRt1/detection_loss‚4.678953091846779 0loss/SRqqt1/detection_loss‚4.363365395649059 0loss/FBt1/assignment_loss‚2.7684538635340603 /loss/FBt2/assignment_loss‚2.794589857260386 1loss/SRqqt4/detection_loss‚2.8841814113159976N‹³JÃÄ 0loss/SRqqt3/detection_loss‚4.130848969991614 % _timestamp‚1.7885290833077836e+09 0loss/SRqqt4/assignment_loss‚5.44988735516866 .loss/FRt3/detection_loss‚4.234842791088989 %loss/total_loss‚116.8059185939568 /loss/FRt3/assignment_loss‚7.250747737430391 /loss/FRt4/assignment_loss‚7.376896398408072 .loss/FRt2/detection_loss‚4.092388374606768 1loss/SRqqt2/detection_loss‚3.2992478782425123 .loss/FBt2/detection_loss‚4.498782691028383 _runtime‚ 67.462099078 /loss/FBt4/detection_loss‚7.1976646346890405 /loss/FRt2/assignment_loss‚7.398452451494005 0loss/FBt3/assignment_loss‚2.7419240053962257 .loss/FBt3/detection_loss‚5.263868105762145 0loss/FBt4/assignment_loss‚2.7755148825438125 trainer/global_step‚49 .loss/FRt1/detection_loss‚4.678953091846779 0loss/SRqqt1/detection_loss‚4.363365395649059 0loss/FBt1/assignment_loss‚2.7684538635340603 /loss/FBt2/assignment_loss‚2.794589857260386 1loss/SRqqt4/detection_loss‚2.8841814113159976 _step‚3 epoch‚0 /loss/FRt1/assignment_loss‚7.348953074879116 1loss/SRqqt1/assignment_loss‚5.410686583132358 1loss/SRqqt2/assignment_loss‚5.447135302485252 .loss/FRt4/detection_loss‚6.180651525301593 .loss/FBt1/detection_loss‚3.801165404475548 1loss/SRqqt3/assignment_loss‚5.416716808226051GMï‘Ä‚(:† “ëÔ¶ßî— gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼5ÓgÅ‚(:Ü “ëÔÕÁÕ› disk.sda3.in‚0( proc.memory.availableMB‚ 2362.0078125) proc.memory.percent‚17.90471076965332 disk.dm-1.out‚ 2.52734375 disk.drbd1260.in‚0% disk./.usageGB‚1176.3351135253906 disk.dm-0.in‚ 178.5234375 disk.dm-1.in‚0 disk.sda3.out‚ 2.52734375$ memory_percent‚85.58344841003418" proc.memory.rssMB‚ 2933.5078125 network.recv‚112333 disk.sda2.out‚0 cpu‚25.363792666722613 proc.cpu.threads‚17 network.sent‚108451) disk./.usagePercent‚78.94483647959083! disk.nvme0c0n1.in‚ 178.5234375' disk.nvme0c0n1.out‚1348.18798828125" disk.dm-0.out‚1348.18798828125 disk.sda.out‚ 2.52734375 disk.sda2.in‚0 disk.nvme0n1.in‚ 178.5234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.01171875 disk.sda.in‚0% disk.nvme0n1.out‚1348.18798828125F­èãÆ‚(:… Ñ“ëÔŒÍÙ– gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.989' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.7978 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³Ú0ãRÇ‚(:Ç Ñ“ëÔ„ž™ disk.nvme0n1.out‚ 1616.265625 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 178.5234375) proc.memory.percent‚17.90471076965332 cpu‚25.331443601487212 network.sent‚146396 disk.sda3.out‚ 3.1015625 disk.dm-0.in‚ 178.5234375$ memory_percent‚85.58640480041504 proc.cpu.threads‚17) disk./.usagePercent‚78.94638558118052$ disk./.usageGB‚1176.358196258545 disk.dm-1.out‚ 3.1015625 disk.sda1.in‚0 disk.sda.out‚ 3.1015625 disk.dm-0.out‚ 1616.265625" proc.memory.rssMB‚ 2933.5078125 network.recv‚119405 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚0.015625 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.out‚ 1616.265625( proc.memory.availableMB‚ 2361.5234375 disk.drbd1260.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 178.5234375ä 2”È‚(:„ à“ëÔßá‘— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.684 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vä$,NÉ‚(:à à“ëÔáž‚™ disk.dm-1.in‚0 disk.nvme0n1.in‚ 199.77734375 disk.sda3.in‚0$ memory_percent‚85.58659553527832) proc.memory.percent‚17.90473461151123 cpu‚25.308691929684105 proc.cpu.threads‚17 network.sent‚153601 disk.nvme0c0n1.out‚ 1855.5625 disk.dm-0.in‚ 199.77734375 disk.dm-0.out‚ 1855.5625( proc.memory.availableMB‚ 2361.4921875) disk./.usagePercent‚78.94638686121819% disk./.usageGB‚1176.3582153320312 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚3.453125 disk.sda2.out‚0 disk.nvme0n1.out‚ 1855.5625" disk.nvme0c0n1.in‚ 199.77734375 disk.dm-1.out‚3.453125 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.015625 disk.sda2.in‚0 disk.sda3.out‚3.453125# proc.memory.rssMB‚ 2933.51171875 network.recv‚123341«—=Ê‚(:… ï“ëÔ¦óë– gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.645' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.929 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oKóúeË‚(:Ú ï“ëÔ°‘ê™ disk.sda.out‚ 3.75390625! proc.memory.rssMB‚ 2933.515625) proc.memory.percent‚17.90475845336914% disk./.usageGB‚1176.3582038879395! disk.dm-0.out‚2144.5673828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.01953125 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚2144.5673828125 disk.drbd1260.in‚0 network.sent‚160307 network.recv‚128583) disk./.usagePercent‚78.94638609319558" disk.nvme0c0n1.in‚ 306.27734375 disk.dm-1.in‚0$ memory_percent‚85.58549880981445' proc.memory.availableMB‚ 2361.671875 disk.sda.in‚0 disk.nvme0n1.in‚ 306.27734375 disk.sda3.in‚0 disk.sda3.out‚ 3.75390625 cpu‚25.288365070964264 proc.cpu.threads‚17& disk.nvme0c0n1.out‚2144.5673828125 disk.dm-0.in‚ 306.27734375 disk.dm-1.out‚ 3.75390625Áv"D0Ì+ _wandbruntime‚128  _runtime‚128 §ù-Âͼ & lr-AdamW/pg1‚6.686891650651293e-05 & lr-AdamW/pg2‚6.686891650651293e-05 trainer/global_step‚99 % _timestamp‚1.7885291439140275e+09 _runtime‚ 128.067583725 _step‚4Á-bÆÎÀ & lr-AdamW/pg2‚6.686891650651293e-05 trainer/global_step‚99 % _timestamp‚1.7885291439140275e+09 _runtime‚ 128.067583725 _step‚4 & lr-AdamW/pg1‚6.686891650651293e-05nk©ÿ0Ï+ _wandbruntime‚128  _runtime‚128J]ßùOБO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2689298 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 195359416832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 199489351168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 195592318464 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4385009757184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3766624684240 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 194805497804 _runtime‚ 128.068970529 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1839991 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309035008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 199204256256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 195592318464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 195034471340 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4504441 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1088920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3961676194940 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1787904 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4141696922112 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 195359416832 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3946355508224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4141947826688 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2664452 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1788589 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚5 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1839989 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3766641723600 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4141696922112 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4529289 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2689298 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4384700722176 trainer/global_step‚99 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0  Ü ›'Ñ•O UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2689298 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 195359416832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 199489351168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 195592318464 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4385009757184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3766624684240 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 194805497804 _runtime‚ 128.068970529 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1839991 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309035008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 199204256256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 195592318464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 195034471340 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4504441 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1088920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3961676194940 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1787904 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4141696922112 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 195359416832 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3946355508224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4141947826688 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2664452 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1788589 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚5 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1839989 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3766641723600 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4141696922112 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4529289 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2689298 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4384700722176 trainer/global_step‚99¯¯”0Ò+ _wandbruntime‚129  _runtime‚129è¡&Ó›M =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1858266 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 197556357632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3804172081024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚99 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4227180435968 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 196992960534 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1858264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4182946984960 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1858264 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2716016 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1806031 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚705621 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4428376909824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2716016 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4183197902848 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1099722 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2691144 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 196763986790 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1099720 _runtime‚ 129.063299599 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4549408 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4183197902848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4227204376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2691144 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4574282 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4001165041558 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 197323442688 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1805341 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4549408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3804155041664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚6 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4183197902848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4227204376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2691144 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4574282 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4001165041558 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 197323442688 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1805341 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4549408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3804155041664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚6 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3804172081024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚99 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4227180435968 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 196992960534 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1858264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4182946984960 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1858264 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2716016 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1806031 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚705621 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4428376909824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2716016 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4183197902848 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1099722 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDevi{î…1|ceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2691144 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 196763986790 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1099720 _runtime‚ 129.063299599 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4549408 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Õ3±Q0Õ+ _wandbruntime‚129  _runtime‚1291FßJÖÄ 1loss/SRqqt2/assignment_loss‚5.352541228999263 % _timestamp‚1.7885291449112577e+09 0loss/FRt2/assignment_loss‚7.3230129938859205 0loss/FBt4/assignment_loss‚2.7892452478408813 .loss/FBt3/detection_loss‚5.599111524410546 trainer/global_step‚99 1loss/SRqqt1/assignment_loss‚5.456188484474465 0loss/SRqqt1/detection_loss‚6.043997979826397 1loss/SRqqt4/detection_loss‚1.8201613874753584 1loss/SRqqt3/assignment_loss‚5.343530086048863 1loss/SRqqt4/assignment_loss‚5.480007377830711 /loss/SRqqt2/detection_loss‚3.59679537995354 /loss/FBt1/detection_loss‚3.4485236725602975 .loss/FBt2/detection_loss‚6.575239048554347 /loss/FRt4/assignment_loss‚7.375471784954979 /loss/FBt3/assignment_loss‚2.473968170583248 &loss/total_loss‚133.59225909015544 0loss/FBt1/assignment_loss‚2.6126681127046285 .loss/FRt1/detection_loss‚4.517120294052029 .loss/FRt3/detection_loss‚3.040028850354676 .loss/FRt4/detection_loss‚4.028676824555511 /loss/FBt4/detection_loss‚27.657239710291226 _runtime‚ 129.064745926 _step‚7 /loss/FRt3/assignment_loss‚7.230539996668978 0loss/FBt2/assignment_loss‚2.5990518239828257 /loss/FRt2/detection_loss‚3.3469776276212473 0loss/SRqqt3/detection_loss‚2.792424563122423 epoch‚0 /loss/FRt1/assignment_loss‚7.089736919403077ïƒÂÃN×È /loss/FBt1/detection_loss‚3.4485236725602975 .loss/FBt2/detection_loss‚6.575239048554347 /loss/FRt4/assignment_loss‚7.375471784954979 /loss/FBt3/assignment_loss‚2.473968170583248 &loss/total_loss‚133.59225909015544 0loss/FBt1/assignment_loss‚2.6126681127046285 .loss/FRt1/detection_loss‚4.517120294052029 .loss/FRt3/detection_loss‚3.040028850354676 .loss/FRt4/detection_loss‚4.028676824555511 /loss/FBt4/detection_loss‚27.657239710291226 _runtime‚ 129.064745926 _step‚7 /loss/FRt3/assignment_loss‚7.230539996668978 0loss/FBt2/assignment_loss‚2.5990518239828257 /loss/FRt2/detection_loss‚3.3469776276212473 0loss/SRqqt3/detection_loss‚2.792424563122423 epoch‚0 /loss/FRt1/assignment_loss‚7.089736919403077 1loss/SRqqt2/assignment_loss‚5.352541228999263 % _timestamp‚1.7885291449112577e+09 0loss/FRt2/assignment_loss‚7.3230129938859205 0loss/FBt4/assignment_loss‚2.7892452478408813 .loss/FBt3/detection_loss‚5.599111524410546 trainer/global_step‚99 1loss/SRqqt1/assignment_loss‚5.456188484474465 0loss/SRqqt1/detection_loss‚6.043997979826397 1loss/SRqqt4/detection_loss‚1.8201613874753584 1loss/SRqqt3/assignment_loss‚5.343530086048863 1loss/SRqqt4/assignment_loss‚5.480007377830711 /loss/SRqqt2/detection_loss‚3.59679537995354 …ÛØ‚(:… þ“ëԸߺ› gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.705' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.141 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Âm+cÙ‚(:Ø þ“ëÔÿǯ  disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚2403.7822265625 disk.dm-0.in‚ 306.27734375 disk.dm-1.out‚ 4.2578125 disk.sda1.in‚0( proc.memory.availableMB‚ 2361.4609375) disk./.usagePercent‚78.94638890927847% disk./.usageGB‚1176.3582458496094 disk.dm-1.in‚0 disk.drbd1260.in‚0) proc.memory.percent‚17.90480613708496 cpu‚25.272768008035325 network.sent‚176798 network.recv‚132398 disk.sda.out‚ 4.2578125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 306.27734375& disk.nvme0c0n1.out‚2403.7822265625 disk.sda1.out‚0 disk.drbd1260.out‚ 0.01953125 disk.nvme0n1.in‚ 306.27734375 disk.sda3.out‚ 4.2578125! disk.dm-0.out‚2403.7822265625# memory_percent‚85.5867862701416" proc.memory.rssMB‚ 2933.5234375 proc.cpu.threads‚17 disk.sda.in‚0x£e•œÚ‚(:‘ ”ëÔ”áì– gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}9¨]Û‚(:Ò ”ëÔÃý°™ disk.sda.in‚0% disk.nvme0c0n1.out‚2643.955078125 disk.dm-1.in‚0 disk.dm-1.out‚ 4.95703125 proc.memory.rssMB‚ 2933.53125) proc.memory.percent‚17.90485382080078 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 4.95703125 disk.sda2.out‚0 disk.nvme0n1.in‚ 306.27734375 disk.sda3.out‚ 4.95703125 disk.dm-0.in‚ 306.27734375% proc.memory.availableMB‚ 2361.1875 network.recv‚139283) disk./.usagePercent‚78.94118043599435 disk.drbd1260.out‚ 0.0234375# disk.nvme0n1.out‚2643.955078125 network.sent‚216208 disk.sda2.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 306.27734375 disk.dm-0.out‚2643.955078125$ memory_percent‚85.58845520019531 cpu‚25.26081203443989 proc.cpu.threads‚17% disk./.usageGB‚1176.2806358337402 disk.sda1.out‚0A[Ã&‘Ü‚(:† œ”ëÔøÐý– gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6062 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸qáoeÝ‚(:Ú œ”ëÔ‚Ùýš disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0n1.out‚2974.029296875) proc.memory.percent‚17.90487766265869 cpu‚25.247166432114586 network.recv‚143173) disk./.usagePercent‚78.95019574131263$ memory_percent‚85.58995723724365# proc.memory.rssMB‚ 2933.53515625 disk.dm-0.out‚2974.029296875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.0234375 disk.sda.out‚ 5.44921875 disk.sda3.in‚0 disk.sda3.out‚ 5.44921875) proc.memory.availableMB‚ 2360.94140625 proc.cpu.threads‚17 network.sent‚219759% disk./.usageGB‚1176.4149703979492 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 306.27734375" disk.nvme0c0n1.in‚ 306.27734375% disk.nvme0c0n1.out‚2974.029296875 disk.dm-0.in‚ 306.27734375 disk.dm-1.out‚ 5.44921875 disk.sda1.in‚0¶ *C›Þ‚(: «”ëÔ±Âì– gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚124.62' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.924000000000003 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±áCÛdß‚(:Ù «”ëÔ¤›ß™ disk.sda.in‚0 disk.sda.out‚ 5.83203125 disk.dm-1.in‚0 disk.dm-1.out‚ 5.83203125 disk.sda1.out‚0$ memory_percent‚85.59308052062988 disk.nvme0n1.in‚ 434.91796875# disk.nvme0n1.out‚3213.587890625 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1260.out‚ 0.02734375) proc.memory.percent‚17.90487766265869 proc.cpu.threads‚17 network.sent‚228203% disk./.usageGB‚1176.4149932861328 disk.sda3.out‚ 5.83203125% disk.nvme0c0n1.out‚3213.587890625 disk.dm-0.in‚ 434.91796875 disk.drbd1260.in‚0) disk./.usagePercent‚78.95019727735783 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 434.91796875 disk.dm-0.out‚3213.587890625( proc.memory.availableMB‚ 2360.4296875# proc.memory.rssMB‚ 2933.53515625 cpu‚25.23955914580602 network.recv‚148220î Éˆ0à+ _wandbruntime‚189  _runtime‚189^‚ÙpÅá¿ _runtime‚ 189.025554379 _step‚8 ' lr-AdamW/pg1‚0.00010064109656030734 ' lr-AdamW/pg2‚0.00010064109656030734 trainer/global_step‚149 % _timestamp‚1.7885292048721266e+09jÕ Éâà ' lr-AdamW/pg2‚0.00010064109656030734 trainer/global_step‚149 % _timestamp‚1.7885292048721266e+09 _runtime‚ 189.025554379 _step‚8 ' lr-AdamW/pg1‚0.00010064109656030734A½B30ã+ _wandbruntime‚189  _runtime‚189U‰Àš'ä”O ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 299280522240 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2753739 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6568975490560 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6204214478336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5643142555440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 293575123456 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6204465382912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 292743538312 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4027250 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5935886093752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5643159594800 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6269694968320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6204465382912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5910639354880 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 293575123456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5910639354880 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2753741 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1048067 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚9 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5910657357824 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2677085 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5936132106648 trainer/global_step‚149 % _timestamp‚1.7885292048730414e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268317696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 293808025088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2677774 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6269718908416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6780991 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1629018 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 292972511848 _runtime‚ 189.026660521 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4027250 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6204214478336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292257792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6756143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2753739 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1048754 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2753741 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2677774 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6269718908416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6780991 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1629018 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 292972511848 _runtime‚ 189.026660521 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4027250 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6204214478336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292257792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6756143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2753739 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1048754 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2753741 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6204214478336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5643142555440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 293575123456 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6204465382912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 292743538312 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4027250 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5935886093752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5643159594800 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6269694968320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6204465382912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5910639354880 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 293575123456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5910639354880 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2753741 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1048067 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚9 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5910657357824 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2677085 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5936132106648 ²Mÿ0æ+ _wandbruntime‚190  _runtime‚190©Ö’ù çŸM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5949943394816 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4029096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2772016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1639820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1055329 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885292058926234e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 295772064256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2695149 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6311378938368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 294931001042 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocaYn3U¬ted‚6825984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6825984 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2772016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2695851 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4053968 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301557999616 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5975374940162 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5949925391872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301275015168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4029096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚10 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6801110 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6245715459072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6612960878080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4053968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273nÃè©&è£M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 295772064256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2695149 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6311378938368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 294931001042 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6825984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6825984 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2772016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2695851 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4053968 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301557999616 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5975374940162 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5949925391872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301275015168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4029096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚10 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6801110 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6245715459072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6612960878080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4053968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5949943394816 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4029096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2772016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1639820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1055329 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885292058926234e+09 _ÓyÇ0é+ _wandbruntime‚190  _runtime‚190UArãJêÄ 0loss/SRqqt1/detection_loss‚6.173857432145339 1loss/SRqqt4/detection_loss‚2.4763987500434896 .loss/FBt3/detection_loss‚6.834521093047583 epoch‚0 /loss/FRt4/assignment_loss‚7.500189537472195 1loss/SRqqt1/assignment_loss‚5.183969571040227 .loss/FRt1/detection_loss‚6.246432415076664 % _timestamp‚1.7885292058953319e+09 _runtime‚ 190.048675724 .loss/FRt3/assignment_loss‚7.12608350476911 1loss/SRqqt2/assignment_loss‚5.106831394542347 /loss/FBt1/assignment_loss‚0.942228078078001 .loss/FRt2/detection_loss‚3.977863586611218 1loss/SRqqt2/detection_loss‚3.0013114956969567 1loss/SRqqt3/detection_loss‚2.6365513012546007 .loss/FBt2/detection_loss‚8.020692999873843 1loss/SRqqt3/assignment_loss‚5.153643764433314 0loss/FBt2/assignment_loss‚0.7265810909725371 .loss/FRt3/detection_loss‚2.233691416140045 .loss/FBt1/detection_loss‚2.537204128082335 &loss/total_loss‚264.19099767560596 /loss/FRt2/assignment_loss‚7.281576813591851 1loss/SRqqt4/assignment_loss‚5.311918900563167 .loss/FBt4/detection_loss‚162.8439510166645 /loss/FRt1/assignment_loss‚7.194045180366153 trainer/global_step‚149 _step‚11 0loss/FBt3/assignment_loss‚0.5184942241758108 0loss/FBt4/assignment_loss‚1.5091086626052856 .loss/FRt4/detection_loss‚3.653851318359375“ƒéäNëÈ .loss/FRt2/detection_loss‚3.977863586611218 1loss/SRqqt2/detection_loss‚3.0013114956969567 1loss/SRqqt3/detection_loss‚2.6365513012546007 .loss/FBt2/detection_loss‚8.020692999873843 1loss/SRqqt3/assignment_loss‚5.153643764433314 0loss/FBt2/assignment_loss‚0.7265810909725371 .loss/FRt3/detection_loss‚2.233691416140045 .loss/FBt1/detection_loss‚2.537204128082335 &loss/total_loss‚264.19099767560596 /loss/FRt2/assignment_loss‚7.281576813591851 1loss/SRqqt4/assignment_loss‚5.311918900563167 .loss/FBt4/detection_loss‚162.8439510166645 /loss/FRt1/assignment_loss‚7.194045180366153 trainer/global_step‚149 _step‚11 0loss/FBt3/assignment_loss‚0.5184942241758108 0loss/FBt4/assignment_loss‚1.5091086626052856 .loss/FRt4/detection_loss‚3.653851318359375 0loss/SRqqt1/detection_loss‚6.173857432145339 1loss/SRqqt4/detection_loss‚2.4763987500434896 .loss/FBt3/detection_loss‚6.834521093047583 epoch‚0 /loss/FRt4/assignment_loss‚7.500189537472195 1loss/SRqqt1/assignment_loss‚5.183969571040227 .loss/FRt1/detection_loss‚6.246432415076664 % _timestamp‚1.7885292058953319e+09 _runtime‚ 190.048675724 .loss/FRt3/assignment_loss‚7.12608350476911 1loss/SRqqt2/assignment_loss‚5.106831394542347 /loss/FBt1/assignment_loss‚0.942228078078001 žŽ?ì‚(:‚ º”ëÔ˜ÆÞ— gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤-ƒ£aí‚(:Ö º”ëÔèáÀ™" proc.memory.rssMB‚ 2933.5390625) disk./.usagePercent‚78.94886629418731 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0n1.out‚3349.845703125 disk.sda.out‚ 6.44140625 cpu‚25.2294758031408% disk.nvme0c0n1.out‚3349.845703125 disk.dm-0.in‚ 434.91796875 disk.dm-1.out‚ 6.44140625$ memory_percent‚85.59060096740723( proc.memory.availableMB‚ 2360.8359375 proc.cpu.threads‚17 network.recv‚152079 disk.sda3.out‚ 6.44140625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.02734375( proc.memory.percent‚17.9049015045166 network.sent‚244736% disk./.usageGB‚1176.3951606750488 disk.nvme0n1.in‚ 434.91796875 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 434.91796875 disk.dm-0.out‚3349.845703125 disk.sda.in‚0rŒØ‘î‚(:† É”ëԜش– gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚127.607' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5214 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@vÛ¾iï‚(:Þ É”ëÔ×…™% disk.nvme0n1.out‚3781.56396484375 disk.sda3.out‚ 7.18359375 disk.sda1.in‚0* proc.memory.percent‚17.905497550964355 cpu‚25.224726738636072 network.sent‚284113) disk./.usagePercent‚78.95044125253796 disk.drbd1260.in‚0 disk.drbd1260.out‚0.03125 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 434.91796875' disk.nvme0c0n1.out‚3781.56396484375 disk.dm-0.in‚ 434.91796875 disk.dm-1.in‚0 disk.dm-1.out‚ 7.18359375 disk.sda.out‚ 7.18359375$ memory_percent‚85.59129238128662# proc.memory.rssMB‚ 2933.63671875 network.recv‚158510 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0" disk.dm-0.out‚3781.56396484375 disk.sda1.out‚0 disk.sda.in‚0) proc.memory.availableMB‚ 2360.72265625$ disk./.usageGB‚1176.418628692627 disk.nvme0n1.in‚ 434.91796875c’Åœð‚(:‘ Ø”ëÔ¾‘Á– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.433' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.286600000000004 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£!–\hñ‚(:Ý Ø”ëÔìÙ disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2933.63671875* proc.memory.percent‚17.905497550964355 proc.cpu.threads‚17 network.recv‚162548% disk./.usageGB‚1176.4810676574707" disk.dm-0.out‚4109.56396484375 disk.sda1.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚4109.56396484375 network.sent‚289455( disk./.usagePercent‚78.9546315838584" disk.nvme0c0n1.in‚ 434.91796875 disk.drbd1260.out‚0.03125 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚85.59136390686035 cpu‚25.216569621941098' disk.nvme0c0n1.out‚4109.56396484375 disk.dm-1.in‚0 disk.dm-1.out‚ 7.86328125 disk.drbd1260.in‚0 disk.sda.out‚ 7.86328125 disk.nvme0n1.in‚ 434.91796875 disk.sda3.out‚ 7.86328125( proc.memory.availableMB‚ 2360.7109375 disk.dm-0.in‚ 434.917968751°Æê‘ò‚(:† ç”ëÔ˜¤²š gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2922 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑK&ºgó‚(:Ü ç”ëÔ…ûûŸ* proc.memory.percent‚17.905783653259277' disk.nvme0c0n1.out‚4347.04052734375$ memory_percent‚85.59179306030273' proc.memory.availableMB‚ 2360.640625 cpu‚25.210763435390586 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.03515625) disk./.usagePercent‚78.95882729133706! disk.nvme0c0n1.in‚ 587.2890625 disk.dm-1.out‚ 8.28515625 disk.sda2.out‚0 disk.sda3.out‚ 8.28515625 network.recv‚167648" disk.dm-0.out‚4347.04052734375 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 8.28515625 disk.sda2.in‚0 disk.nvme0n1.in‚ 587.2890625% disk.nvme0n1.out‚4347.04052734375 network.sent‚294267$ disk./.usageGB‚1176.543586730957 disk.dm-0.in‚ 587.2890625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2933.68359375sl¢Ñ0ô+ _wandbruntime‚249  _runtime‚249#\†—ÆõÀ _step‚12 ' lr-AdamW/pg1‚0.00013441327661410174 ' lr-AdamW/pg2‚0.00013441327661410174 trainer/global_step‚199 % _timestamp‚1.7885292656036937e+09 _runtime‚ 249.757387062TäR ÊöÄ % _timestamp‚1.7885292656036937e+09 _runtime‚ 249.757387062 _step‚12 ' lr-AdamW/pg1‚0.00013441327661410174 ' lr-AdamW/pg2‚0.00013441327661410174 trainer/global_step‚199 Üq)j0÷+ _wandbruntime‚249  _runtime‚249_ÖC ‘ø”O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚13 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7910588018356 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5340356 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _runtime‚ 249.759324674 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2169120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8353917410816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 $ _timestamp‚1.788529265605327e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 391790830080 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 392023731712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7874941204480 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3569044 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8753366946304 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 399473475584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8753671787008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8266732034560 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3667489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1400595 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7519677466000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390681578820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7874959207424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.ìXKF current‚671 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5365202 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8266732034560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8266982939136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3667489 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9007845 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8266982939136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7519660426640 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 390910552356 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229šÞ'ù˜O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7519660426640 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 390910552356 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚13 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7910588018356 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5340356 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _runtime‚ 249.759324674 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2169120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8353917410816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 $ _timestamp‚1.788529265605327e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 391790830080 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 392023731712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7874941204480 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3569044 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8753366946304 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 399473475584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8753671787008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8266732034560 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3667489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1400595 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7519677466000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390681578820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7874959207424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5365202 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8266732034560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8266982939136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3667489 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9007845 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8266982939136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 v™éû0ú+ _wandbruntime‚251  _runtime‚251jP;£¥&ûŸM TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 393754855936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 393987770880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 401753943040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1407730 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚199 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3685764 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7914227241472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7950076864974 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8308233015296 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚723 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7949830851870 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚14 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7914245244416 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 393754855936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9052812 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5367048 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8307982097408 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3685766 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2179922 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7557207823424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3586929 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8797355323904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8395601380864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9077686 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7557190784064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3685766 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚ 251.105294858 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7914245244416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3685764 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8797046302208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3587652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8395577440768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7885292669506123e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7914227241472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582"a&ü£M XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7914227241472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 393754855936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 393987770880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 401753943040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1407730 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚199 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3685764 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7914227241472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7950076864974 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8308233015296 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚723 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7949830851870 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚14 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7914245244416 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 393754855936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9052812 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5367048 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8307982097408 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3685766 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2179922 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7557207823424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3586929 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8797355323904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8395601380864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9077686 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7557190784064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3685766 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚ 251.105294858 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7914245244416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3685764 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8797046302208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3587652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8395577440768 >6DeviceStatsMonitor.on_train_batÙolLHch_end/num_device_alloc‚582 % _timestamp‚1.7885292669506123e+09 ¯÷>0ý+ _wandbruntime‚251  _runtime‚251œ†ê¿SþÍ 1loss/SRqqt1/assignment_loss‚3.669453499164987 /loss/FBt3/detection_loss‚3.0266315293859463 _runtime‚ 251.107334332 0loss/FBt4/assignment_loss‚0.8428544156253338 /loss/FRt1/assignment_loss‚7.248672236566958 /loss/FBt4/detection_loss‚10.600276637077332 2loss/SRqqt3/assignment_loss‚3.4538890246687264 .loss/FRt4/detection_loss‚4.141448635517097 epoch‚0 _step‚15 $loss/total_loss‚98.3541587053113 .loss/FRt1/detection_loss‚4.827028614023458 /loss/FRt2/assignment_loss‚7.290514808434707 /loss/FRt4/assignment_loss‚7.470580007971787 1loss/SRqqt2/assignment_loss‚3.638030936501243 /loss/FRt3/detection_loss‚3.0258511270989072 1loss/SRqqt2/detection_loss‚2.6808730155229568 .loss/FBt2/detection_loss‚9.354831129312515 trainer/global_step‚199 1loss/FBt3/assignment_loss‚0.15641617544033393 1loss/SRqqt4/assignment_loss‚3.793476963043213 1loss/FBt2/assignment_loss‚0.44289642774189514 1loss/SRqqt1/detection_loss‚3.2657191740705613 /loss/FRt2/detection_loss‚3.2403389943333774 1loss/SRqqt3/detection_loss‚2.3832466117716553 1loss/SRqqt4/detection_loss‚2.4380889965327692 /loss/FBt1/detection_loss‚3.4411361728442205 % _timestamp‚1.7885292669539237e+09 /loss/FRt3/assignment_loss‚7.285337439450351 0loss/FBt1/assignment_loss‚0.6365661332109833KÔþWÿÑ .loss/FRt4/detection_loss‚4.141448635517097 epoch‚0 _step‚15 $loss/total_loss‚98.3541587053113 .loss/FRt1/detection_loss‚4.827028614023458 /loss/FRt2/assignment_loss‚7.290514808434707 /loss/FRt4/assignment_loss‚7.470580007971787 1loss/SRqqt2/assignment_loss‚3.638030936501243 /loss/FRt3/detection_loss‚3.0258511270989072 1loss/SRqqt2/detection_loss‚2.6808730155229568 .loss/FBt2/detection_loss‚9.354831129312515 trainer/global_step‚199 1loss/FBt3/assignment_loss‚0.15641617544033393 1loss/SRqqt4/assignment_loss‚3.793476963043213 1loss/FBt2/assignment_loss‚0.44289642774189514 1loss/SRqqt1/detection_loss‚3.2657191740705613 /loss/FRt2/detection_loss‚3.2403389943333774 1loss/SRqqt3/detection_loss‚2.3832466117716553 1loss/SRqqt4/detection_loss‚2.4380889965327692 /loss/FBt1/detection_loss‚3.4411361728442205 % _timestamp‚1.7885292669539237e+09 /loss/FRt3/assignment_loss‚7.285337439450351 0loss/FBt1/assignment_loss‚0.6365661332109833 1loss/SRqqt1/assignment_loss‚3.669453499164987 /loss/FBt3/detection_loss‚3.0266315293859463 _runtime‚ 251.107334332 0loss/FBt4/assignment_loss‚0.8428544156253338 /loss/FRt1/assignment_loss‚7.248672236566958 /loss/FBt4/detection_loss‚10.600276637077332 2loss/SRqqt3/assignment_loss‚3.4538890246687264äÓëg€‚(:… ö”ëÔ¼«·— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.208' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0416 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾böHa‚(:Ö ö”ëÔ¹¹™š" disk.dm-0.out‚4631.11083984375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.03515625 disk.sda2.in‚0 disk.sda3.out‚8.671875 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 587.2890625$ memory_percent‚85.59551239013672# proc.memory.rssMB‚ 2933.69140625 proc.cpu.threads‚17 disk.sda.out‚8.671875 network.sent‚299641 network.recv‚171227) disk./.usagePercent‚78.96722433846016% disk./.usageGB‚1176.6687088012695 disk.dm-1.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 2360.03125 disk.dm-0.in‚ 587.2890625 disk.dm-1.out‚8.671875 disk.sda1.out‚0 disk.nvme0n1.in‚ 587.2890625% disk.nvme0n1.out‚4631.11083984375' disk.nvme0c0n1.out‚4631.11083984375* proc.memory.percent‚17.905831336975098 cpu‚25.206096323637293Î룡›‚‚(: …•ëÔšâÙ– gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.712799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­!Afƒ‚(:Û …•ëÔ‰Û÷™) disk./.usagePercent‚78.96564810007183 disk.sda2.in‚0% disk.nvme0n1.out‚4772.77001953125 disk.sda3.out‚ 9.08984375 disk.dm-1.in‚0 disk.sda1.in‚0$ memory_percent‚85.59255599975586' proc.memory.availableMB‚ 2360.515625$ disk./.usageGB‚1176.645221710205 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 9.08984375 disk.sda3.in‚0 disk.dm-1.out‚ 9.08984375 disk.sda1.out‚0 cpu‚25.200128222020982 disk.sda2.out‚0 disk.nvme0n1.in‚ 587.2890625" disk.dm-0.out‚4772.77001953125 proc.cpu.threads‚17 network.sent‚351827 network.recv‚177472 disk.drbd1260.out‚ 0.0390625! disk.nvme0c0n1.in‚ 587.2890625' disk.nvme0c0n1.out‚4772.77001953125 disk.dm-0.in‚ 587.2890625# proc.memory.rssMB‚ 2933.69140625* proc.memory.percent‚17.905831336975098½Ïœ„‚(:‘ ”•ëÔ–°Ð– gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.888' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.777600000000003 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ùŠœf…‚(:Û ”•ëԌ܊™$ memory_percent‚85.59408187866211 disk.sda1.out‚0 disk.drbd1260.out‚ 0.0390625 disk.sda.in‚0 disk.nvme0n1.in‚ 587.2890625' proc.memory.availableMB‚ 2360.265625 cpu‚25.19636920428079 proc.cpu.threads‚17 network.sent‚355145 disk.sda3.out‚ 9.61328125! disk.nvme0c0n1.in‚ 587.2890625' disk.nvme0c0n1.out‚5176.94970703125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda.out‚ 9.61328125 disk.sda2.out‚0) disk./.usagePercent‚78.96564835607937 disk.sda3.in‚0 disk.dm-0.in‚ 587.2890625" disk.dm-0.out‚5176.94970703125 disk.drbd1260.in‚0% disk.nvme0n1.out‚5176.94970703125# proc.memory.rssMB‚ 2933.69140625* proc.memory.percent‚17.905831336975098 network.recv‚181212% disk./.usageGB‚1176.6452255249023 disk.dm-1.out‚ 9.61328125 disk.sda2.in‚0ÌV£‘†‚(:† £•ëÔŸ—ç– gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9546 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝŠ™ i‡‚(:Þ £•ëÔåÅ™( proc.memory.availableMB‚ 2361.2109375% disk.nvme0n1.out‚5571.37255859375 network.sent‚361578 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.04296875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚186137% disk./.usageGB‚1176.9186210632324 disk.sda1.out‚0 disk.sda.out‚ 9.95703125 disk.nvme0n1.in‚ 763.3203125 disk.sda3.in‚0 disk.sda3.out‚ 9.95703125! disk.nvme0c0n1.in‚ 763.3203125# proc.memory.rssMB‚ 2933.69140625* proc.memory.percent‚17.905831336975098 cpu‚25.191252844219186 proc.cpu.threads‚17) disk./.usagePercent‚78.98399616005082' disk.nvme0c0n1.out‚5571.37255859375 disk.dm-0.in‚ 763.3203125" disk.dm-0.out‚5571.37255859375 disk.dm-1.in‚0 disk.dm-1.out‚ 9.95703125$ memory_percent‚85.58831214904785°Œ³0ˆ+ _wandbruntime‚311  _runtime‚311*|_ʼn¿ ' lr-AdamW/pg1‚0.00016818545666789616 ' lr-AdamW/pg2‚0.00016818545666789616 trainer/global_step‚249 $ _timestamp‚1.788529327590066e+09 _runtime‚ 311.743736939 _step‚16`Ë™ ÉŠÃ ' lr-AdamW/pg1‚0.00016818545666789616 ' lr-AdamW/pg2‚0.00016818545666789616 trainer/global_step‚249 $ _timestamp‚1.788529327590066e+09 _runtime‚ 311.743736939 _step‚16‘8¾0‹+ _wandbruntime‚311  _runtime‚311ŒÌÿÿ§'Œ¡O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 _runtime‚ 311.745990798 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4459092 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11259547 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6678308 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9396195337200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1304 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10937912062464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11284395 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6703154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2709218 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 490239438336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 499796149248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚249 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4581241 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1749872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10438115913216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10438091973120 % _timestamp‚1.7885293275919015e+09 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6678308 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10329249590784 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 488848592864 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9839243054080 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1285 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4581239 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2709220 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10329500495360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚17 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 490239438336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10329249590784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 499508957184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4581241 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9839243054080 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1285 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4581239 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2709220 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10329500495360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚17 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 490239438336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10329249590784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 499508957184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4581241 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6705000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 492203477504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9878547094016 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 491970562560 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6729872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 491970562560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11329388 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4599514 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2720022 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 490807082058 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4599516 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4476166 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10370750571520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11304514 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9878529091072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4599514 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9878529091072 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4476804 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4599516 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6705000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9878547094016 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 501552414720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10370750571520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270401536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10370499653632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10479775943168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10981622699520 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 490578108314 _step‚18 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 501822816256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11304514 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2720020;ÜÈÁµ&¯M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4476166 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10370750571520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11304514 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9878529091072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4599514 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9878529091072 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4476804 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4599516 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6705000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9878547094016 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 501552414720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10370750571520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270401536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10370499653632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10479775943168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10981622699520 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 490578108314 _step‚18 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 501822816256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11304514 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2720020 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6705000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 492203477504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9878547094016 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 491970562560 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6729872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 491970562560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11329388 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4599514 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2720022 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 490807082058 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4599516 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5346661 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12522314415616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3249318 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11859499841772 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494991 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 588455144960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12391767147008 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5494991 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8041106 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12392018051584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11272713208400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11803544903680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 586786633372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2097343 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _step‚21 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 588222243328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2098054 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚ 371.814089577 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8041106 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11803562906624 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13536097 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13122055387136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12391767147008 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 599451682304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12522290475520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5347374 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5494989 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13121742157824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5494989 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11803544903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3249320 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 599740971520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5346661 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12522314415616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3249318 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11859499841772 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494991 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 588455144960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12391767147008 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5494991 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8041106 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12392018051584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11272713208400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11803544903680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 586786633372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2097343 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _step‚21 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 588222243328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2098054 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚ 371.814089577 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8041106 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11803562906624 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13536097 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13122055387136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12391767147008 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 599451682304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12522290475520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5347374 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5494989 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944ºr€0¢+ _wandbruntime‚372  _runtime‚372?úšª¹&£³M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2103972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 trainer/global_step‚299 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11310243565824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 590419184128 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12433017209856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚ 372.810990581 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11842848943616 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 590186269184 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8042952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5364092 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5513264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚22 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11310226526464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 590186269184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12563974445568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3260122 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 590419184128 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5513266 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13165425120768 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 601450675200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 588516148822 % _timestamp‚1.7885293886566749e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3260120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11842848943616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13556216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12563998385664 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5513264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12433268127744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13581090 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12433268127744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13581090 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2104658 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5513266 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8067824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8042952 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 590419184128 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5513266 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13165425120768 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 601450675200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 588516148822 % _timestamp‚1.7885293886566749e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3260120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11842848943616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13556216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12563998385664 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5513264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12433268127744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13581090 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12433268127744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13581090 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2104658 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5513266 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8067824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8042952 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11310243565824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 590419184128 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12433017209856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚ 372.810990581 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11842848943616 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 590186269184 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8042952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5364092 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5513264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ÀDE0¥+ _wandbruntime‚372  _runtime‚372<¸)¦Í /loss/FRt2/detection_loss‚2.8783870906683435 /loss/FBt1/detection_loss‚1.8958679213158967 .loss/FBt3/detection_loss‚2.807435555340798 epoch‚0 _step‚23 /loss/FRt1/assignment_loss‚7.371534690856934 /loss/FRt4/assignment_loss‚7.484874165058136 2loss/SRqqt1/assignment_loss‚3.4611519778647075 1loss/SRqqt4/assignment_loss‚3.844299349784851 0loss/FBt1/assignment_loss‚0.7831739181934227 .loss/FRt1/detection_loss‚4.043578367233277 1loss/SRqqt1/detection_loss‚3.7906032735254707 trainer/global_step‚299 1loss/SRqqt2/assignment_loss‚3.536865219473839 2loss/SRqqt3/assignment_loss‚3.3155491716331906 0loss/FBt4/assignment_loss‚0.7613673840338985 1loss/SRqqt4/detection_loss‚3.1872595642879604 /loså]Rs/FBt2/detection_loss‚13.916810477321798 /loss/FBt4/detection_loss‚12.915280451377233 &loss/total_loss‚104.69174886020241 _runtime‚ 372.813083657 1loss/FBt3/assignment_loss‚0.27821993867644557 0loss/SRqqt2/detection_loss‚5.021966960281134 $ _timestamp‚1.788529388659692e+09 /loss/FRt3/assignment_loss‚7.271922296093356 0loss/FBt2/assignment_loss‚0.8762032202317972 .loss/FRt3/detection_loss‚2.687102056739311 /loss/FRt2/assignment_loss‚7.322358516224644 /loss/FRt4/detection_loss‚3.9500706885010004 1loss/SRqqt3/detection_loss‚1.2898666054849552UÔW§Ñ /loss/FBt1/detection_loss‚1.8958679213158967 .loss/FBt3/detection_loss‚2.807435555340798 epoch‚0 _step‚23 /loss/FRt1/assignment_loss‚7.371534690856934 /loss/FRt4/assignment_loss‚7.484874165058136 2loss/SRqqt1/assignment_loss‚3.4611519778647075 1loss/SRqqt4/assignment_loss‚3.844299349784851 0loss/FBt1/assignment_loss‚0.7831739181934227 .loss/FRt1/detection_loss‚4.043578367233277 1loss/SRqqt1/detection_loss‚3.7906032735254707 trainer/global_step‚299 1loss/SRqqt2/assignment_loss‚3.536865219473839 2loss/SRqqt3/assignment_loss‚3.3155491716331906 0loss/FBt4/assignment_loss‚0.7613673840338985 1loss/SRqqt4/detection_loss‚3.1872595642879604 /loss/FBt2/detection_loss‚13.916810477321798 /loss/FBt4/detection_loss‚12.915280451377233 &loss/total_loss‚104.69174886020241 _runtime‚ 372.813083657 1loss/FBt3/assignment_loss‚0.27821993867644557 0loss/SRqqt2/detection_loss‚5.021966960281134 $ _timestamp‚1.788529388659692e+09 /loss/FRt3/assignment_loss‚7.271922296093356 0loss/FBt2/assignment_loss‚0.8762032202317972 .loss/FRt3/detection_loss‚2.687102056739311 /loss/FRt2/assignment_loss‚7.322358516224644 /loss/FRt4/detection_loss‚3.9500706885010004 1loss/SRqqt3/detection_loss‚1.2898666054849552 /loss/FRt2/detection_loss‚2.8783870906683435‘ ;‘¨‚(:† î•ëÔʸ¤— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÉò†e©‚(:Ú î•ëÔî£þ™% disk./.usageGB‚1176.9235877990723 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.out‚ 12.24609375 disk.dm-0.in‚ 893.140625$ memory_percent‚85.59284210205078 disk.dm-1.out‚ 12.24609375 disk.drbd1260.in‚0 disk.sda2.in‚0" disk.dm-0.out‚7053.56494140625" proc.memory.rssMB‚ 2933.6953125* proc.memory.percent‚17.905855178833008 cpu‚25.17637147333739 disk.sda2.out‚0) disk./.usagePercent‚78.98432948186041 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.05078125' disk.nvme0c0n1.out‚7053.56494140625& proc.memory.availableMB‚ 2360.46875 network.sent‚447343 disk.sda.out‚ 12.24609375 disk.nvme0n1.in‚ 893.140625% disk.nvme0n1.out‚7053.56494140625 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 893.140625 proc.cpu.threads‚17 network.recv‚208778ø[*盪‚(: ý•ëÔçíÒ– gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.71' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.541999999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#¿e^«‚(:Ó ý•ëÔ‚Ñ£™ proc.cpu.threads‚17 network.recv‚214471( disk./.usagePercent‚78.9843346020111% disk.nvme0c0n1.out‚7482.041015625 disk.dm-0.out‚7482.041015625 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 12.6953125 disk.nvme0c0n1.in‚ 893.140625 disk.dm-0.in‚ 893.140625 disk.dm-1.in‚0 disk.dm-1.out‚ 12.6953125 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 12.6953125 disk.sda2.out‚0% disk./.usageGB‚1176.9236640930176 disk.drbd1260.out‚ 0.0546875# disk.nvme0n1.out‚7482.041015625) proc.memory.availableMB‚ 2360.87109375" proc.memory.rssMB‚ 2933.6953125* proc.memory.percent‚17.905855178833008 cpu‚25.174945171084666 network.sent‚485680 disk.sda2.in‚0 disk.nvme0n1.in‚ 893.140625$ memory_percent‚85.590386390686045[f:‘¬‚(:† Œ–ëÔÖÔ– gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3172 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EjcÑ^­‚(:Ó Œ–ëÔ›‡Ã™ disk.sda3.in‚0 disk.sda3.out‚ 13.30859375 disk.drbd1260.in‚0 cpu‚25.171799903884658 network.recv‚218117) disk./.usagePercent‚78.98433613805629 disk.sda.out‚ 13.30859375 disk.sda2.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚490927 disk.nvme0c0n1.in‚ 893.140625% disk.nvme0c0n1.out‚7779.498046875 disk.dm-0.in‚ 893.140625 disk.dm-0.out‚7779.498046875 disk.dm-1.out‚ 13.30859375 disk.sda1.in‚0# memory_percent‚85.5916976928711% disk./.usageGB‚1176.9236869812012 disk.nvme0n1.in‚ 893.140625# disk.nvme0n1.out‚7779.498046875 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.0546875& proc.memory.availableMB‚ 2360.65625" proc.memory.rssMB‚ 2933.6953125* proc.memory.percent‚17.905855178833008 disk.sda.in‚0 disk.sda2.out‚0ÒÜux‘®‚(:† ›–ëÔƺÀ– gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0106 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z«Èi¯‚(:Þ ›–ëÔ§§ä˜ disk.sda2.out‚0" proc.memory.rssMB‚ 2933.6953125* proc.memory.percent‚17.905855178833008 network.sent‚495530$ disk.nvme0n1.out‚7915.2177734375 disk.sda2.in‚0$ memory_percent‚85.59191226959229) proc.memory.availableMB‚ 2360.62109375 cpu‚25.169523472116754 proc.cpu.threads‚17 network.recv‚223172) disk./.usagePercent‚78.98433613805629 disk.nvme0n1.in‚ 908.9453125! disk.nvme0c0n1.in‚ 908.9453125& disk.nvme0c0n1.out‚7915.2177734375! disk.dm-0.out‚7915.2177734375 disk.dm-1.out‚ 13.65234375 disk.drbd1260.in‚0% disk./.usageGB‚1176.9236869812012 disk.sda3.in‚0 disk.dm-0.in‚ 908.9453125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 13.65234375 disk.sda3.out‚ 13.65234375 disk.drbd1260.out‚ 0.05859375åsø50°+ _wandbruntime‚432  _runtime‚432˜>úƱÀ trainer/global_step‚349 % _timestamp‚1.7885294481793716e+09 _runtime‚ 432.333059128 _step‚24 ' lr-AdamW/pg1‚0.00023572981677548498 ' lr-AdamW/pg2‚0.00023572981677548498œ1 Ê²Ä ' lr-AdamW/pg1‚0.00023572981677548498 ' lr-AdamW/pg2‚0.00023572981677548498 trainer/global_step‚349 % _timestamp‚1.7885294481793716e+09 _runtime‚ 432.333059128 _step‚24JnsŽ0³+ _wandbruntime‚432  _runtime‚432NÈý¯'´©O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14454284703232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15787799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 699750796800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9354212 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15305944194048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6408741 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2441093 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14454284703232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2441798 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14454535607808 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6408739 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 686437949952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13833709740584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6230511 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3789418 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9379058 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚25 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13767864756224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14606512918016 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13767864756224 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 686670851584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14454284703232 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15787799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 699750796800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9354212 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15305944194048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6408741 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2441093 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14454284703232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2441798 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14454535607808 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6408739 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 686437949952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13833709740584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6230511 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3789418 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9379058 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚25 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13767864756224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14606512918016 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 71345111044þz90¶+ _wandbruntime‚433  _runtime‚433åk ·³M YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13807132790272 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6248121 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13807132790272 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13873198586994 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15349935037440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13873444600098 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 701738149376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 686683163074 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6248786 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 688634890752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14648172947968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3800220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15807918 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3800222 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ €.r:688401975808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 688634890752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13186744397664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9405776 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2447901 % _timestamp‚1.7885294492858589e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14648196888064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13186761437024 trainer/global_step‚349 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 688401975808 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15349630210048 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14495785683968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15832792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2448564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9380904 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 701457262080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14495534766080 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15807918 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3800222 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 688401975808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 688634890752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13186744397664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9405776 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2447901 % _timestamp‚1.7885294492858589e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14648196888064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13186761437024 trainer/global_step‚349 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 688401975808 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15349630210048 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14495785683968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15832792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2448564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9380904 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 701457262080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14495534766080 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6248121 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13807132790272 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13873198586994 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15349935037440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13873444600098 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 701738149376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 686683163074 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6248786 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 688634890752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14648172947968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3800220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15808411665188 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16517053164032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7118390 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17490571146752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15732166605824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788529502416288e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 799859726336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15732148602880 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 486.569951547 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7322489 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7322491 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18039501 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7322489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10717010 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_po‚ýëol.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7117703 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 799576728576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7322491 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16516802259456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 782662714388Kо¶'É°O XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16517053164032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10717010 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16516802259456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10692164 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 486.569951547 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7322489 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7322491 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18039501 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7322489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10717010 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7117703 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 799576728576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7322491 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16516802259456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 782662714388 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _step‚29 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15732166605824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15732148602880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 782433740852 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10692164 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15808411665188 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16517053164032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7118390 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17490571146752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15732166605824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788529502416288e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 799859726336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15732148602880 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15847900511806 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 _step‚30 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 786850597376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4340320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16732395390464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10718856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16732371450368 % _timestamp‚1.7885295033744237e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚15771452642816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17533948062208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 801576611840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7340764 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2794983 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17534250792448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10743728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18059620 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 784621203582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278790144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 784392229838 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15771452642816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16558303240192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18084494 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15771434639872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 302730240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚399 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18084494 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 786617682432 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16558303240192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15063262268864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 801855401984 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4340322 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7135303 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18059620 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 784621203582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278790144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 784392229838 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15771452642816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16558303240192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18084494 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15771434639872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 302730240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚399 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18084494 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 786617682432 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16558303240192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15063262268864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 801855401984 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4340322 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7135303 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15847900511806 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 _step‚30 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 786850597376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4340320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16732395390464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10718856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16732371450368 % _timestamp‚1.7885295033744237e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚15771452642816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17533948062208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 801576611840` §0Í+ _wandbruntime‚487  _runtime‚487I,Ô•LÎÆ _runtime‚ 487.530748973 _step‚31 .loss/FBt1/assignment_loss‚0.83677503983596 .loss/FRt1/detection_loss‚3.753844742117257 /loss/FRt3/assignment_loss‚7.156799270556523 /loss/FRt4/assignment_loss‚7.470569807907631 2loss/SRqqt1/assignment_loss‚3.3623372077941895 epoch‚0 % _timestamp‚1.7885295033774161e+09 1loss/SRqqt4/assignment_loss‚3.610192712984587 /loss/FBt1/detection_loss‚1.8467790863632312 /loss/FBt3/detection_loss‚1.9903778805913144 trainer/global_step‚399 2loss/SRqqt3/assignment_loss‚3.2641777992248535 .loss/FRt4/detection_loss‚4.914897834432536 0loss/SRqqt1/detection_loss‚2.605794076486067 %loss/total_loss‚88.77705300030728 0loss/SRqqt2/detection_loss‚2.611685527521267 /loss/FBt2/detection_loss‚12.723987543582917 /loss/FRt1/assignment_loss‚7.279121596237709 /loss/FBt4/assignment_loss‚0.765251680935866 1loss/SRqqt4/detection_loss‚1.9076069048500781 .loss/FBt4/detection_loss‚4.057420739957265 .loss/FRt2/assignment_loss‚7.28441681581385 0loss/FBt3/assignment_loss‚0.2031960833523861 .loss/FRt2/detection_loss‚3.145698781472211 .loss/FRt3/detection_loss‚2.880567028151395 2loss/SRqqt2/assignment_loss‚3.3784727757437185 0loss/FBt2/assignment_loss‚0.4999118220701348 1loss/SRqqt3/detection_loss‚1.2271702423243336³ØýøýÏÊ 2loss/SRqqt2/assignment_loss‚3.3784727757437185 0loss/FBt2/assignment_loss‚0.4999118220701348 1loss/SRqqt3/detection_loss‚1.2271702423243336 _runtime‚ 487.530748973 _step‚31 .loss/FBt1/assignment_loss‚0.83677503983596 .æ{œSloss/FRt1/detection_loss‚3.753844742117257 /loss/FRt3/assignment_loss‚7.156799270556523 /loss/FRt4/assignment_loss‚7.470569807907631 2loss/SRqqt1/assignment_loss‚3.3623372077941895 epoch‚0 % _timestamp‚1.7885295033774161e+09 1loss/SRqqt4/assignment_loss‚3.610192712984587 /loss/FBt1/detection_loss‚1.8467790863632312 /loss/FBt3/detection_loss‚1.9903778805913144 trainer/global_step‚399 2loss/SRqqt3/assignment_loss‚3.2641777992248535 .loss/FRt4/detection_loss‚4.914897834432536 0loss/SRqqt1/detection_loss‚2.605794076486067 %loss/total_loss‚88.77705300030728 0loss/SRqqt2/detection_loss‚2.611685527521267 /loss/FBt2/detection_loss‚12.723987543582917 /loss/FRt1/assignment_loss‚7.279121596237709 /loss/FBt4/assignment_loss‚0.765251680935866 1loss/SRqqt4/detection_loss‚1.9076069048500781 .loss/FBt4/detection_loss‚4.057420739957265 .loss/FRt2/assignment_loss‚7.28441681581385 0loss/FBt3/assignment_loss‚0.2031960833523861 .loss/FRt2/detection_loss‚3.145698781472211 .loss/FRt3/detection_loss‚2.880567028151395„¿oûœÐ‚(:‘ æ–ëÔ¶Ó– gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚141.243' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.248600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}Ê|)mÑ‚(:â æ–ëÔ¼’™! disk.nvme0n1.in‚ 1183.56640625" disk.dm-0.out‚9685.77197265625 disk.drbd1260.out‚ 0.06640625 disk.sda.in‚0$ memory_percent‚85.59274673461914 disk.sda3.in‚0 disk.dm-0.in‚ 1183.56640625 disk.dm-1.out‚ 16.3359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 2360.484375) disk./.usagePercent‚78.91879257712503 disk.sda3.out‚ 16.3359375# disk.nvme0c0n1.in‚ 1183.56640625' disk.nvme0c0n1.out‚9685.77197265625 disk.drbd1260.in‚0* proc.memory.percent‚17.905855178833008 cpu‚25.156848733453806 proc.cpu.threads‚17% disk./.usageGB‚1175.9470405578613% disk.nvme0n1.out‚9685.77197265625 disk.dm-1.in‚0 disk.sda.out‚ 16.3359375" proc.memory.rssMB‚ 2933.6953125 network.sent‚581352 network.recv‚246383Kˆ4è‘Ò‚(:† õ–ëÔïǯ– gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚118.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7516 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RGjÓ‚(:ß õ–ëÔ¾¹¦˜ disk.sda.in‚0! disk.nvme0n1.in‚ 1183.56640625 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 1183.56640625 disk.dm-0.in‚ 1183.56640625 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.91879001704969 disk.drbd1260.in‚0# disk.nvme0n1.out‚9779.263671875 disk.sda3.out‚ 16.73046875 disk.dm-1.out‚ 16.73046875$ memory_percent‚85.59203147888184 cpu‚25.154449761042283 disk.sda2.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2933.6953125* proc.memory.percent‚17.905855178833008 proc.cpu.threads‚17 network.sent‚618752 network.recv‚252807 disk.drbd1260.out‚ 0.0703125 disk.sda.out‚ 16.73046875 disk.sda2.out‚0% disk.nvme0c0n1.out‚9779.263671875 disk.dm-0.out‚9779.263671875( proc.memory.availableMB‚ 2360.6015625% disk./.usageGB‚1175.9470024108887£ñÝö‘Ô‚(:† „—ëÔ¾›ž— gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚108.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6596 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üå;RgÕ‚(:Ü „—ëÔ޾ɚ# disk.nvme0c0n1.in‚ 1183.56640625% disk.nvme0c0n1.out‚9992.673828125 disk.sda1.out‚0# memory_percent‚85.5936050415039 disk.sda.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1183.56640625# disk.nvme0n1.out‚9992.673828125 disk.dm-0.out‚9992.673828125 disk.dm-1.out‚ 17.29296875& proc.memory.availableMB‚ 2360.34375* proc.memory.percent‚17.905855178833008 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0703125 disk.sda2.out‚0 disk.sda3.out‚ 17.29296875 disk.dm-0.in‚ 1183.56640625 proc.cpu.threads‚17% disk./.usageGB‚1175.9352645874023 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2933.6953125 cpu‚25.151891446808325 network.sent‚624145 network.recv‚256537) disk./.usagePercent‚78.91800228186683 disk.sda.out‚ 17.29296875G£š\0Ö+ _wandbruntime‚538  _runtime‚538­ë"¨Ä×¾ & lr-AdamW/pg2‚0.0003032741768830738 trainer/global_step‚449 % _timestamp‚1.7885295547329326e+09 _runtime‚ 538.886616848 _step‚32 & lr-AdamW/pg1‚0.0003032741768830738f‡]3ÈØ & lr-AdamW/pg2‚0.0003032741768830738 trainer/global_step‚449 % _timestamp‚1.7885295547329326e+09 _runtime‚ 538.886616848 _step‚32 & lr-AdamW/pg1‚0.0003032741768830738 —®d0Ù+ _wandbruntime‚538  _runtime‚538?‚œ³'Ú­O YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19674483127808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 882869363200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12030116 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 272512000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18579319815680 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4869620 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8236241 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17696450452480 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 899597145088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17696450452480 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8006680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4869618 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 880600754896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _step‚33 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16902249782640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12054962 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12054962 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18774885982720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 538.887537234 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17696468455424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3136356 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚17782621564000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8236239 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20266355 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296452096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12030116 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 882869363200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20291203 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19674779579904‚ŒUí·'Û±O K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8236239 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20266355 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296452096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12030116 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 882869363200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20291203 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19674779579904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19674483127808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 882869363200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12030116 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 272512000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18579319815680 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4869620 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8236241 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17696450452480 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 899597145088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17696450452480 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8006680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4869618 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 880600754896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _step‚33 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16902249782640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12054962 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12054962 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18774885982720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 538.887537234 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17696468455424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3136356 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚17782621564000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0!éR§Ó0Ü+ _wandbruntime‚539  _runtime‚539¬B/ÂݶM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4880422 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19718468392960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 882330270346 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12056808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20336196 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12081680 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8023995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16939780140064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18620820796416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20311322 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18816593892864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18620820796416 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8254516 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚ 539.882145729 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17735736489472 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 901589418496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20336196 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4880420 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 884833389056 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 884833389056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18816569952768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚449 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8024681 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 885066304000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 885066304000 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16939797179424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3143575 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 882559244090 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚3×:=·ú 4 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17822110410410 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901874500096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17735736489472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12056808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19718159371264 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12081680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8254514 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17735754492416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18620569878528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20336196 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12081680 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8023995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16939780140064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18620820796416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20311322 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18816593892864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18620820796416 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8254516 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚ 539.882145729 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17735736489472 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 901589418496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20336196 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4880420 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 884833389056 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 884833389056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18816569952768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚449 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8024681 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 885066304000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 885066304000 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16939797179424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3143575 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 882559244090 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚34 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17822110410410 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901874500096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17735736489472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12056808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19718159371264 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12081680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8254514 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17735754492416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18620569878528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20641837371904 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3484790 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19660770305024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21858765867520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22542905 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 981317971456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5409720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 999958088704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9149991 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8894508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5409718õ÷ëíSï±O C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22518057 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20642088276480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 981317971456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5409720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 999958088704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9149991 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8894508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5409718 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19660752302080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19757323488604 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 _step‚37 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8895184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 981085069824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20641837371904 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 981317971456 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 981085069824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20859084485120 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13392914 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13392914 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3485464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 300646400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20641837371904 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3484790 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19660770305024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21858765867520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22542905 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885296147120726e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9168266 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19700038339072 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9168266 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 983282010624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3491954 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19700056342016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18816298011264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8911820 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1001960907776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19700038339072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21902446378496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19796812335222 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8912476 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22587898 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13394760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22563024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20900768455168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19700056342016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21902753303040 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22587898 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0½ òÃ&ò½M WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 983049095680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13419632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20900792395264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 980497284598 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885296147120726e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9168266 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19700038339072 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9168266 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 983282010624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3491954 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19700056342016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18816298011264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8911820 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1001960907776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19700038339072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21902446378496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19796812335222 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8912476 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22587898 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13394760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22563024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20900768455168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19700056342016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21902753303040 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22587898 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3491300 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19796566322118 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13394760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 O0+ _wandbruntime‚653  _runtime‚653‰ðW‰¾'‚¸O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22704605832704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21731533387416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3831013 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21625054151680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 _runtime‚ 653.61152455 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1079300776448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 _step‚41 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1079533678080 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22704354928128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14706020 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 300646400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14706020 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21625072154624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273/ÊcœÂ'ƒ¼O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24769759 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7885296694578218e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1076476835912 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24043278319104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21625054151680 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22704354928128 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10063741 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10063739 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22943282987520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10063739 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5949818 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1079533678080 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22943306927616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22704605832704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21731533387416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3831013 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21625054151680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 _runtime‚ 653.61152455 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1079300776448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 _step‚41 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1079533678080 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22704354928128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14706020 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 300646400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14706020 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21625072154624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3836970 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24086966894080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14757584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24839600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14732712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9798302 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1101688817664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21771022233826 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1081264802304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9797590 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21771268246930 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10082014 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20692832921824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21664358191616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14757584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24814726 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24839600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24086655775232 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10082016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21664358191616 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1101975996416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDevicSz ßeStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885296704865556e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1081497717248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1078206351362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10082016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22984990897664 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20692815882464 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚549 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9797590 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21771268246930 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10082014 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20692832921824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21664358191616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14757584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24814726 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24839600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24086655775232 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10082016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21664358191616 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1101975996416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885296704865556e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1081497717248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1078206351362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10082016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22984990897664 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20692815882464 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚549 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3836970 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24086966894080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14757584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24839600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14732712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9798302 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1101688817664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21771022233826 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1081264802304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920*~;v0‡+ _wandbruntime‚654  _runtime‚654ÂÖ#QˆË 2loss/SRqqt1/assignment_loss‚2.9599582967550857 1loss/SRqqt1/detection_loss‚2.8497615501932474 0loss/SRqqt4/detection_loss‚4.157426958734339 trainer/global_step‚549 1loss/FBt3/assignment_loss‚0.45869321699877447 0loss/FBt4/assignment_loss‚0.3965833376952153 1loss/SRqqt2/detection_loss‚2.5089183492319926 /loss/FRt2/assignment_loss‚5.793627931225684 1loss/SRqqt3/assignment_loss‚2.973218960421426 _step‚43 /loss/FRt4/assignment_loss‚6.696076147896903 /loss/FRt2/detection_loss‚2.3992266058921814 1loss/SRqqt3/detection_loss‚1.1266881161081117 .loss/FBt1/detection_loss‚6.146537172794342 $ _timestamp‚1.788529670489222e+09 1loss/SRqqt2/assignment_loss‚1.795100721777702 2loss/SRqqt4/assignment_loss‚3.2840650515122847 0loss/FBt2/assignment_loss‚0.7460062205791473 /loss/FRt1/detection_loss‚3.1176764082908632 .loss/FBt2/detection_loss‚20.71417808532715 epoch‚0 _runtime‚ 654.642565461 /loss/FRt3/assignment_loss‚5.430634448605199 .loss/FRt4/detection_loss‚3.242008734600885 .loss/FBt4/detection_loss‚3.415437244270977 %loss/total_loss‚91.35588852998904 /loss/FRt1/assignment_loss‚6.598224258422851 0loss/FBt1/assignment_loss‚0.5725352249417331 /loss/FRt3/detection_loss‚2.4941304451034916 /loss/FBt3/detection_loss‚1.4791750426094656Šu´U‰Ï .loss/FBt1/detection_loss‚6.146537172794342 $ _timestamp‚1.788529670489222e+09 1loss/SRqqt2/assignment_loss‚1.795100721777702 2loss/SRqqt4/assignment_loss‚3.2840650515122847 0loss/FBt2/assignment_loss‚0.7460062205791473 /loss/FRt1/detection_loss‚3.1176764082908632 .loss/FBt2/detection_loss‚20.71417808532715 epoch‚0 _runtime‚ 654.642565461 /loss/FRt3/assignment_loss‚5.430634448605199 .loss/FRt4/detection_loss‚3.242008734600885 .loss/FBt4/detection_loss‚3.415437244270977 %loss/total_loss‚91.35588852998904 /loss/FRt1/assignment_loss‚6.598224258422851 0loss/FBt1/assignment_loss‚0.5725352249417331 /loss/FRt3/detection_loss‚2.4941304451034916 /loss/FBt3/detection_loss‚1.4791750426094656 2loss/SRqqt1/assignment_loss‚2.9599582967550857 1loss/SRqqt1/detection_loss‚2.8497615501932474 0loss/SRqqt4/detection_loss‚4.157426958734339 trainer/global_step‚549 1loss/FBt3/assignment_loss‚0.45869321699877447 0loss/FBt4/assignment_loss‚0.3965833376952153 1loss/SRqqt2/detection_loss‚2.5089183492319926 /loss/FRt2/assignment_loss‚5.793627931225684 1loss/SRqqt3/assignment_loss‚2.973218960421426 _step‚43 /loss/FRt4/assignment_loss‚6.696076147896903 /loss/FRt2/detection_loss‚2.3992266058921814 1loss/SRqqt3/detection_loss‚1.1266881161081117+×é}‘Š‚(:† ‹˜ëÔËóª— gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.059' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2118 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦O‡ír‹‚(:ç ‹˜ëÔÏÙ›š disk.sda.out‚ 21.91015625 disk.sda2.out‚0 disk.sda3.out‚ 21.91015625 disk.dm-0.in‚ 1509.8515625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚85.59730052947998# proc.memory.rssMB‚ 2933.75390625 proc.cpu.threads‚17% disk./.usageGB‚1175.9106712341309 disk.nvme0n1.in‚ 1509.8515625& disk.nvme0n1.out‚12507.28369140625) proc.memory.percent‚17.90621280670166 cpu‚25.138621445676222 network.sent‚765039 network.recv‚299447) disk./.usagePercent‚78.91635180129359( disk.nvme0c0n1.out‚12507.28369140625# disk.dm-0.out‚12507.28369140625 disk.drbd1260.out‚ 0.08984375 disk.sda2.in‚0) proc.memory.availableMB‚ 2359.73828125" disk.nvme0c0n1.in‚ 1509.8515625 disk.dm-1.out‚ 21.91015625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0TãbŒ‚(:… š˜ëÔÙ„— gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3082 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÎ?‹u‚(:ê š˜ëÔ´¶Àš disk.dm-1.out‚ 22.26953125 disk.sda1.out‚0 network.sent‚813698) disk./.usagePercent‚78.92444727154133% disk./.usageGB‚1176.0312995910645 disk.sda2.out‚0! disk.nvme0n1.in‚ 1598.83984375 disk.sda3.out‚ 22.26953125# disk.dm-0.out‚12720.71337890625 disk.sda.in‚0# disk.nvme0c0n1.in‚ 1598.83984375( disk.nvme0c0n1.out‚12720.71337890625 disk.dm-0.in‚ 1598.83984375 disk.sda1.in‚0$ memory_percent‚85.59730052947998) proc.memory.availableMB‚ 2359.73828125) proc.memory.percent‚17.90621280670166 cpu‚25.137787316272696& disk.nvme0n1.out‚12720.71337890625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.08984375 disk.sda.out‚ 22.26953125# proc.memory.rssMB‚ 2933.75390625 proc.cpu.threads‚17 network.recv‚304234 disk.sda2.in‚0 disk.sda3.in‚0Óݨ…‘Ž‚(:† ©˜ëԈ梗 gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5542 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º4ªj‚(:ß ©˜ëÔžàœ›# proc.memory.rssMB‚ 2933.75390625) proc.memory.percent‚17.90621280670166 cpu‚25.136954011598235 network.sent‚820296) disk./.usagePercent‚78.88529527130737% disk./.usageGB‚1175.4479064941406 disk.dm-1.in‚0 disk.dm-1.out‚ 22.7265625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.09375" disk.dm-0.out‚12938.6201171875 disk.sda1.in‚0 disk.sda.out‚ 22.7265625 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1625.69921875% disk.nvme0n1.out‚12938.6201171875 disk.sda3.in‚0 network.recv‚309441' disk.nvme0c0n1.out‚12938.6201171875 disk.sda.in‚0 disk.sda3.out‚ 22.7265625$ memory_percent‚85.59579849243164 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1625.69921875 disk.dm-0.in‚ 1625.69921875' proc.memory.availableMB‚ 2359.984375[ì⟜‚(:‘ ¸˜ëÔ÷㢖 gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚140.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.060399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eœß(_‘‚(:Ô ¸˜ëÔЭ֘ disk.sda2.in‚0 disk.sda3.out‚23.125' disk.nvme0c0n1.out‚13134.3583984375" disk.dm-0.out‚13134.3583984375$ memory_percent‚85.59739589691162) disk./.usagePercent‚78.88684437289704 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1625.69921875 disk.dm-0.in‚ 1625.69921875) proc.memory.percent‚17.90621280670166 cpu‚25.136048872042466$ disk./.usageGB‚1175.470989227295 disk.sda1.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 1625.69921875 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚825121 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.09375% disk.nvme0n1.out‚13134.3583984375) proc.memory.availableMB‚ 2359.72265625# proc.memory.rssMB‚ 2933.75390625 network.recv‚312584 disk.dm-1.out‚23.125 disk.sda.out‚23.125Çk 0’+ _wandbruntime‚713  _runtime‚713ÚJ†Æ“À ' lr-AdamW/pg2‚0.00040459071704445704 trainer/global_step‚599 % _timestamp‚1.7885297296812289e+09 _runtime‚ 713.834669828 _step‚44 ' lr-AdamW/pg1‚0.00040459071704445704mèn$Ê”Ä % _timestamp‚1.7885297296812289e+09 _runtime‚ 713.834669828 _step‚44 ' lr-AdamW/pg1‚0.00040459071704445704 ' lr-AdamW/pg2‚0.00040459071704445704 trainer/global_step‚599,½]å0•+ _wandbruntime‚713  _runtime‚713[&µÁ'–»O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6489918 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25027481489920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1177516483072 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27046309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23589356001280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16043972 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4179597 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4178908 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27021461 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10977491 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6489920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23705989299124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23589374004224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16068818 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1199970643456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10668826 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24766872484352 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10977491 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10977489 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16043972 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1174185902884 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23589356001280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177516483072 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24767123388928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16068818 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10977489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1199689742848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23589374004224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10669517 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24766872484352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1174414876420 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26227171232768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6489920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23705989299124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 64:{‰M{ 58321920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23589374004224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16068818 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1199970643456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10668826 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24766872484352 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10977491 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10977489 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16043972 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1174185902884 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23589356001280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177516483072 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24767123388928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16068818 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10977489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1199689742848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23589374004224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10669517 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24766872484352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1174414876420 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26227171232768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1177516483072 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27046309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23589356001280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16043972 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4179597 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4178908 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27021461 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10977491 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2-O4‹Û0˜+ _wandbruntime‚714  _runtime‚714B{óèÉ&™ÃM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23628642038272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1201969972736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27091302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10995766 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1179713423872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1176144391870 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22569333753664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4186153 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27066428 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27091302 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788529730660063e+09 _runtime‚ 714.81415575 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6500720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6500722 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1201682793984 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1179480508928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16095536 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10686873©`Í&šÇM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23628642038272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1201969972736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27091302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10995766 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1179713423872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1176144391870 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22569333753664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4186153 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27066428 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27091302 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788529730660063e+09 _runtime‚ 714.81415575 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6500720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6500722 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1201682793984 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1179480508928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16095536 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10686873 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273.à)`0›+ _wandbruntime‚714  _runtime‚714‰=#LœÆ .loss/FBt2/detection_loss‚7.179049581289291 %loss/total_loss‚76.55020753664199 % _timestamp‚1.7885297306628275e+09 0loss/FRt1/assignment_loss‚6.5394442194984075 /loss/FRt4/detection_loss‚3.7047725717226663 1loss/SRqqt3/detection_loss‚0.9575072094345483 .loss/FBt1/detection_loss‚5.199250938800665 _step‚47 /loss/FRt4/assignment_loss‚6.723637086373788 0loss/SRqqt1/assignment_loss‚3.26212610801061 2loss/SRqqt3/assignment_loss‚2.7498335236700897 /loss/FBt2/assignment_loss‚0.645453467965126 .loss/FRt1/detection_loss‚2.802429676055908 /loss/FRt2/detection_loss‚2.6352374632747804 0loss/SRqqt1/detection_loss‚3.634475839138031 .loss/FBt4/detection_loss‚2.756431820780732 _runtime‚ 714.816175718 0loss/FBt3/assignment_loss‚0.5864569076734243 1loss/SRqqt4/assignment_loss‚2.954908914036221 /loss/FBt1/assignment_loss‚1.281240232826139 /loss/FRt3/detection_loss‚2.0896773366392525 /loss/FRt3/assignment_loss‚4.270218227220618 1loss/SRqqt2/assignment_loss‚2.509286977350712 /loss/FBt4/assignment_loss‚0.434364028491092 0loss/SRqqt2/detection_loss‚2.496040011445681 /loss/FBt3/detection_loss‚1.3063411503821842 /loss/FRt2/assignment_loss‚5.876916340419224 1loss/SRqqt4/detection_loss‚3.9551079041428037 trainer/global_step‚599 epoch‚0QaŒØPÊ /loss/FRt4/assignment_loss‚6.723637086373788 0loss/SRqqt1/assignment_loss‚3.26212610801061 2loss/SRqqt3/assignment_loss‚2.7498335236700897 /loss/FBt2/assignment_loss‚0.645453467965126 .loss/FRt1/detection_loss‚2.802429676055908 /loss/FRt2/detection_loss‚2.6352374632747804 0loss/SRqqt1/detection_loss‚3.634475839138031 .loss/FBt4/detection_loss‚2.756431820780732 _runtime‚ 714.816175718 0loss/FBt3/assignment_loss‚0.5864569076734243 1loss/SRqqt4/assignment_loss‚2.954908914036221 /loss/FBt1/assignment_loss‚1.281240232826139 /loss/FRt3/detection_loss‚2.0896773366392525 /loss/FRt3/assignment_loss‚4.270218227220618 1loss/SRqqt2/assignment_loss‚2.509286977350712 /loss/FBt4/assignment_loss‚0.434364028491092 0loss/SRqqt2/detection_loss‚2.496040011445681 /loss/FBt3/detection_loss‚1.3063411503821842 /loss/FRt2/assignment_loss‚5.876916340419224 1loss/SRqqt4/detection_loss‚3.9551079041428037 trainer/global_step‚599 epoch‚0 .loss/FBt2/detection_loss‚7.179049581289291 %loss/total_loss‚76.55020753664199 % _timestamp‚1.7885297306628275e+09 0loss/FRt1/assignment_loss‚6.5394442194984075 /loss/FRt4/detection_loss‚3.7047725717226663 1loss/SRqqt3/detection_loss‚0.9575072094345483 .loss/FBt1/detection_loss‚5.199250938800665 _step‚47/YŸL]ž‚(:„ ǘëÔ‰ïÓ– gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.702 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚Â.esŸ‚(:è ǘëԾ⤙ disk.dm-1.out‚ 23.73046875 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 23.73046875 disk.sda2.out‚0% disk./.usageGB‚1175.4709739685059! disk.nvme0n1.in‚ 1625.69921875 disk.sda3.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 2359.140625) proc.memory.percent‚17.90621280670166 cpu‚25.134960213491485 proc.cpu.threads‚17 network.sent‚844471) disk./.usagePercent‚78.88684334886692( disk.nvme0c0n1.out‚13667.74267578125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.09765625# proc.memory.rssMB‚ 2933.75390625# disk.nvme0c0n1.in‚ 1625.69921875 disk.dm-0.in‚ 1625.69921875# disk.dm-0.out‚13667.74267578125 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚85.60094833374023 network.recv‚317710& disk.nvme0n1.out‚13667.74267578125 disk.sda3.out‚ 23.73046875Ä %‘ ‚(:† Ö˜ëÔ©·¿– gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚137.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.5794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òŒ¹¡‚(:â Ö˜ëÔ?w/ºZØèë˜ disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 1625.69921875 disk.sda1.in‚0& disk.nvme0n1.out‚13867.89501953125( disk.nvme0c0n1.out‚13867.89501953125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0$ proc.memory.availableMB‚2359.875) proc.memory.percent‚17.90621280670166 disk.sda3.out‚ 24.1015625 disk.sda.in‚0 disk.sda.out‚ 24.1015625! disk.nvme0n1.in‚ 1625.69921875# disk.dm-0.out‚13867.89501953125 disk.drbd1260.out‚ 0.09765625$ memory_percent‚85.59646606445312# proc.memory.rssMB‚ 2933.75390625 network.recv‚321394% disk./.usageGB‚1175.4783096313477 disk.dm-1.out‚ 24.1015625 cpu‚25.133662031863675 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1625.69921875 network.sent‚878965) disk./.usagePercent‚78.88733565135526óÂ^y‘¢‚(:† å˜ëÔý¹½• gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.226' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4452 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬é¥Ân£‚(:ã å˜ëÔ²õÓš network.recv‚326086( disk./.usagePercent‚78.8821238499732 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚14065.63525390625 disk.sda3.out‚ 24.6640625 cpu‚25.13353559091263$ disk./.usageGB‚1175.400650024414 disk.drbd1260.out‚ 0.1015625 disk.sda.in‚0! disk.nvme0n1.in‚ 1823.99609375# disk.nvme0c0n1.in‚ 1823.99609375 disk.dm-0.in‚ 1823.99609375 disk.dm-1.out‚ 24.6640625 disk.sda.out‚ 24.6640625 disk.sda2.in‚0 disk.sda3.in‚0# disk.dm-0.out‚14065.63525390625 disk.dm-1.in‚0) proc.memory.percent‚17.90621280670166 proc.cpu.threads‚17 network.sent‚887212 disk.sda1.out‚0( disk.nvme0c0n1.out‚14065.63525390625$ memory_percent‚85.59677600860596) proc.memory.availableMB‚ 2359.82421875# proc.memory.rssMB‚ 2933.75390625pZ,¯‘¤‚(:† ô˜ëÔ±ùË– gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r)km¥‚(:â ô˜ëÔÔ­‡˜ disk.sda2.out‚0! disk.nvme0n1.in‚ 1823.99609375# disk.dm-0.out‚14065.97119140625$ memory_percent‚85.59684753417969 cpu‚25.132764922475324 network.recv‚329553 disk.sda.in‚0& disk.nvme0n1.out‚14065.97119140625( disk.nvme0c0n1.out‚14065.97119140625 disk.dm-0.in‚ 1823.99609375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.1015625# proc.memory.rssMB‚ 2933.75390625) disk./.usagePercent‚78.88665287926145 disk.sda.out‚ 25.0703125 disk.sda3.in‚0 disk.sda3.out‚ 25.0703125# disk.nvme0c0n1.in‚ 1823.99609375 disk.sda1.in‚0) proc.memory.percent‚17.90621280670166 proc.cpu.threads‚17 network.sent‚890200 disk.dm-1.out‚ 25.0703125 disk.sda1.out‚0 disk.drbd1260.in‚0% proc.memory.availableMB‚ 2359.8125% disk./.usageGB‚1175.4681358337402 disk.sda2.in‚0mVÔÕ0¦+ _wandbruntime‚770  _runtime‚770é)Xħ¾ trainer/global_step‚649 % _timestamp‚1.7885297859243124e+09 _runtime‚ 770.077794011 _step‚48 & lr-AdamW/pg1‚0.0004383628970982514 & lr-AdamW/pg2‚0.0004383628970982514Ñç±rȨ & lr-AdamW/pg1‚0.0004383628970982514 & lr-AdamW/pg2‚0.0004383628970982514 trainer/global_step‚649 % _timestamp‚1.7885297859243124e+09 _runtime‚ 770.077794011 _step‚480½7àí0©+ _wandbruntime‚770  _runtime‚770. FÀ'ªºO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17381924 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28411746707968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27111679992320 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25680445210832 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24408338306800 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29298011 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4528286 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25680691223728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1272123943392 _runtime‚ 770.079176979 _step‚49 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275732189696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29273163 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7030018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26829390040576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25553675853824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11891239 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25553657850880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1275965091328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17406770 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 $ _timestamp‚1.788529785925457e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299747194880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26829640945152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11557615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7030020 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11891241 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11891239 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27111703932416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1275732189696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25553675853824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1272352916928 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26829390040576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25553675853824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11891239 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25553657850880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1275965091328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17406770 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 $ _timestamp‚1.788529785925457e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299747194880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26829640945152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11557615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7030020 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11891241 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11891239 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27111703932416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1275732189696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25553675853824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 12723529169281çéZ0¬+ _wandbruntime‚771  _runtime‚771Ù® ­ÅM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1302029762048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27153387902464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1274082432378 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575074 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17408616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1277929130496 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28455112837120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26870891021312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4534254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11575755 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29343004 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11909516 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885297870243328e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17433488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318130 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7040822 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1277696215552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11909516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17408616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25592943887872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4534933 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28455417664512 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1301748874752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1274311406122 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚50 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27153363962368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24445851624864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26870891021312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25592943887872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24445868664224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_enØâÕd/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17433488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11909514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚ 771.179165765 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1277929130496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29343004 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11909516 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885297870243328e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17433488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318130 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7040822 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1277696215552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11909516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17408616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25592943887872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4534933 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28455417664512 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1301748874752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1274311406122 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚50 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27153363962368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24445851624864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26870891021312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25592943887872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24445868664224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17433488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11909514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚ 771.179165765 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1277929130496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575074 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17408616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1277929130496 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28455112837120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26870891021312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4534254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11575755 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18719876 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12448583 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4878463 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚29195878494720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1373947896320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18744722 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27517959700480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311132160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12804991 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12804989 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28892158501376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1373947896320 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28892158501376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28891907596800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12447877 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7570118 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26284839138640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31549713 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27517977703424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29195902434816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31524865 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1374180797952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚699 _runtime‚ 830.561458022 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27517959700480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27517977703424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30595540917760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309‘7ÿQ ¿¿O E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚699 _runtime‚ 830.561458022 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27517959700480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27517977703424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30595540917760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_sËZÃ:»tart/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27655147135436 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18744722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚53 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7570120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1399949615104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18719876 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12448583 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4878463 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚29195878494720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1373947896320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18744722 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27517959700480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311132160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12804991 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12804989 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28892158501376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1373947896320 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28892158501376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28891907596800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12447877 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7570118 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26284839138640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31549713 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27517977703424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29195902434816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31524865 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1374180797952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚5825ŒËCr0À+ _wandbruntime‚831  _runtime‚831;~’ùË&ÁÅM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31594706 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28933408577536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1401652459520 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27694635982054 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18771440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4884366 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31569832 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18746568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1376144837120 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28933408577536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26322386535424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12465286 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885298473849058e+09 _step‚54 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29237586404864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31569832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12823264 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30639214924288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31594706 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 319507456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12823266 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28933157659648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1375911922176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚ 831.539112196 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27557245737472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401948026880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376144837120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12823264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚699 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7580920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27557263740416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12465992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372020472886 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27557245737472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27557263740416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18746568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7580922 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26322369496064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1372249446630 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104¦AÏ&ÂÉM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12465286 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885298473849058e+09 _step‚54 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29237586404864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31569832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12823264 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30639214924288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31594706 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 319507456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12823266 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28933157659648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1375911922176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚ 831.539112196 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27557245737472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401948026880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376144837120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12823264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚699 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7580920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27557263740416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12465992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372020472886 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27557245737472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27557263740416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18746568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7580922 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26322369496064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1372249446630 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31594706 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28933408577536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1401652459520 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27694635982054 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18771440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4884366 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31569832 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18746568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1376144837120 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28933408577536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263223865354246#ÖÈÉ0Ã+ _wandbruntime‚831  _runtime‚831ߧ¯ïPÄÊ trainer/global_step‚699 /loss/FRt2/assignment_loss‚7.297945136115665 /loss/FRt3/assignment_loss‚5.069987986882528 0loss/FBt2/assignment_loss‚0.5252079185884213 1loss/FBt4/assignment_loss‚0.24062001351315607 .loss/FRt2/detection_loss‚3.742262343565623 /loss/FRt3/detection_loss‚1.9590261908372244 .loss/FBt2/detection_loss‚5.479136824607849 2loss/SRqqt4/assignment_loss‚3.0780480666594072 /loss/FRt1/detection_loss‚2.7135703464349112 .loss/FBt3/detection_loss‚2.129699521945286 2loss/SRqqt1/assignment_loss‚3.0451749930014977 $ _timestamp‚1.788529847387813e+09 _step‚55 0loss/FBt3/assignment_loss‚0.6756197580199751 0loss/SRqqt4/detection_loss‚4.071191706440666 %loss/total_loss‚76.50539790500028 epoch‚0 1loss/FBt1/assignment_loss‚0.44910563527105296 /loss/SRqqt1/detection_loss‚3.35125189102613 /loss/FBt4/detection_loss‚1.4966308009149372 0loss/SRqqt2/assignment_loss‚2.70076390675136 2loss/SRqqt3/assignment_loss‚2.4890846396485964 1loss/SRqqt3/detection_loss‚0.9051119309539596 .loss/FBt1/detection_loss‚7.498389363288879 /loss/FRt1/assignment_loss‚6.426943918069203 /loss/FRt4/assignment_loss‚6.112203283129998 /loss/FRt4/detection_loss‚2.3767652646550594 _runtime‚ 831.541227804 1loss/SRqqt2/detection_loss‚2.67165646467890075èÍTÅÎ /loss/FBt4/detection_loss‚1.4966308009149372 0loss/SRqqt2/assignment_loss‚2.70076390675136 2loss/SRqqt3/assignment_loss‚2.4890846396485964 1loss/SRqqt3/detection_loss‚0.9051119309539596 .loss/FBt1/detection_loss‚7.498389363288879 /loss/FRt1/assignment_loss‚6.426943918069203 /loss/FRt4/assignment_loss‚6.112203283129998 /loss/FRt4/detection_loss‚2.3767652646550594 _runtime‚ 831.541227804 1loss/SRqqt2/detection_loss‚2.6716564646789007 trainer/global_step‚699 /loss/FRt2/assignment_loss‚7.297945136115665 /loss/FRt3/assignment_loss‚5.069987986882528 0loss/FBt2/assignment_loss‚0.5252079185884213 1loss/FBt4/assignment_loss‚0.24062001351315607 .loss/FRt2/detection_loss‚3.742262343565623 /loss/FRt3/detection_loss‚1.9590261908372244 .loss/FBt2/detection_loss‚5.479136824607849 2loss/SRqqt4/assignment_loss‚3.0780480666594072 /loss/FRt1/detection_loss‚2.7135703464349112 .loss/FBt3/detection_loss‚2.129699521945286 2loss/SRqqt1/assignment_loss‚3.0451749930014977 $ _timestamp‚1.788529847387813e+09 _step‚55 0loss/FBt3/assignment_loss‚0.6756197580199751 0loss/SRqqt4/detection_loss‚4.071191706440666 %loss/total_loss‚76.50539790500028 epoch‚0 1loss/FBt1/assignment_loss‚0.44910563527105296 /loss/SRqqt1/detection_loss‚3.351251891026137îA ÝœÆ‚(:‘ ¿™ëÔÍÁ–— gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.576' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.115199999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊXpÇ‚(:å ¿™ëÔÈã®™ network.sent‚975436$ disk./.usageGB‚1175.914249420166" disk.nvme0c0n1.in‚ 1967.3203125 disk.dm-0.in‚ 1967.3203125# disk.dm-0.out‚15525.59716796875 disk.dm-1.out‚ 27.53515625 disk.sda2.out‚0 disk.sda3.out‚ 27.53515625( disk./.usagePercent‚78.9165919363607 disk.dm-1.in‚0 disk.drbd1260.in‚0) proc.memory.availableMB‚ 2359.41796875 cpu‚25.129881770895498 network.recv‚351007( disk.nvme0c0n1.out‚15525.59716796875 disk.drbd1260.out‚ 0.11328125 disk.sda.out‚ 27.53515625& disk.nvme0n1.out‚15525.59716796875 disk.sda3.in‚0$ memory_percent‚85.59925556182861# proc.memory.rssMB‚ 2933.75390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1967.3203125) proc.memory.percent‚17.90621280670166 proc.cpu.threads‚17ºÝ­‘È‚(:† ΙëÔ®³‡• gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6662 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ڽɂ(:Þ ΙëÔªµ›& disk.nvme0n1.out‚15778.26513671875 disk.dm-0.in‚ 1967.3203125# disk.dm-0.out‚15778.26513671875# proc.memory.rssMB‚ 2933.75390625 cpu‚25.12955937837474 proc.cpu.threads‚17 network.sent‚ 1.012767e+06( disk./.usagePercent‚78.9417393004416$ disk./.usageGB‚1176.288963317871 disk.sda.in‚0 disk.sda.out‚27.78125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 1967.3203125 disk.dm-1.out‚27.78125 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1171875 disk.sda3.out‚27.78125( disk.nvme0c0n1.out‚15778.26513671875" memory_percent‚85.597825050354) proc.memory.availableMB‚ 2359.65234375) proc.meà h°¬mory.percent‚17.90621280670166 network.recv‚355527 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 1967.3203125M¸Ò›Ê‚(: Ý™ëÔ¨Åú– gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.091' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.818199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÅ íuË‚(:ê Ý™ëÔ  …š disk.sda2.in‚0 disk.nvme0n1.in‚ 2165.0078125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 2165.0078125) disk./.usagePercent‚78.94173546032857 disk.sda2.out‚0& disk.nvme0n1.out‚16176.02294921875 disk.dm-1.out‚ 28.21484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0# memory_percent‚85.5980634689331( disk.nvme0c0n1.out‚16176.02294921875 disk.drbd1260.out‚ 0.1171875) proc.memory.percent‚17.90621280670166 cpu‚25.129130381635047 network.sent‚ 1.017221e+06 disk.sda3.out‚ 28.21484375 disk.dm-0.in‚ 2165.0078125# disk.dm-0.out‚16176.02294921875 disk.dm-1.in‚0) proc.memory.availableMB‚ 2359.61328125# proc.memory.rssMB‚ 2933.75390625 proc.cpu.threads‚17 network.recv‚360334$ disk./.usageGB‚1176.288906097412 disk.sda.in‚0 disk.sda.out‚ 28.21484375qè²ã‘Ì‚(:† ì™ëÔÚÍÜ– gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.199' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0398 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K ä pÍ‚(:å ì™ëÔ˜Õ™ disk.nvme0n1.in‚ 2165.0078125 disk.sda3.out‚ 28.6796875 disk.sda1.in‚0 cpu‚25.128978895308517 proc.cpu.threads‚17% disk.nvme0n1.out‚16504.4365234375 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 2165.0078125 disk.dm-0.in‚ 2165.0078125" disk.dm-0.out‚16504.4365234375 disk.dm-1.out‚ 28.6796875 disk.drbd1260.out‚ 0.12109375) proc.memory.availableMB‚ 2359.34765625 network.recv‚363683 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2933.75390625) proc.memory.percent‚17.90621280670166) disk./.usagePercent‚78.94173981245667% disk./.usageGB‚1176.2889709472656 disk.sda.out‚ 28.6796875' disk.nvme0c0n1.out‚16504.4365234375 disk.sda1.out‚0" memory_percent‚85.599684715271 network.sent‚ 1.024014e+06óÆU0Î+ _wandbruntime‚888  _runtime‚888¾†î\ÄϾ _step‚56 & lr-AdamW/pg1‚0.0005059072572058403 & lr-AdamW/pg2‚0.0005059072572058403 trainer/global_step‚749 % _timestamp‚1.7885299043088539e+09 _runtime‚ 888.462439803³mÑ»ÈР& lr-AdamW/pg1‚0.0005059072572058403 & lr-AdamW/pg2‚0.0005059072572058403 trainer/global_step‚749 % _timestamp‚1.7885299043088539e+09 _runtime‚ 888.462439803 _step‚5686L0Ñ+ _wandbruntime‚888  _runtime‚888CÏ¥Á'Ò»O VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33801415 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13718741 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1499628099584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20057828 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31280076997120 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1472163602944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13718739 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5226861 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33776567 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13718741 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28161357009840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29482261550080 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29482279553024 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8110220 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29629603047144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30954425153024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32779705096704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30954676057600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30954425153024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20082674 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8110218 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5226161 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31280100937216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1468000024408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1499919485952 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚57 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0îÖlÅ'Ó¿O [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1472396504576 % _timestamp‚1.7885299043100672e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1468228997944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29482261550080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29482279553024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13718739 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32780020423168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5226161 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31280100937216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1468000024408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1499919485952 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚57 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33801415 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13718741 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1499628099584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20057828 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31280076997120 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1472163602944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13718739 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5226861 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33776567 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13718741 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28161357009840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29482261550080 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29482279553024 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8110220 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29629603047144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30954425153024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32779705096704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30954676057600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30954425153024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20082674 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8110218 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33846408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31321784907264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1470187487138 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚58 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33821534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32823696210944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31321760967168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1469958513394 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28198887367264 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30995675215872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20084520 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788529905394661e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13737016 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20109392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 trainer/global_step‚749 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13737016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13354457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474360543744 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 315313152 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32823380897792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28198904406624 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33846408 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30995926133760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33821534 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13737014 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8121022 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29521547587072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474127628800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474127628800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13353759 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29521547587072 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚ 889.549018065 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33846408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31321784907264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1470187487138 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚58 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33821534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32823696210944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31321760967168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1469958513394 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28198887367264 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30995675215872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20084520 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788529905394661e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13737016 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20109392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 trainer/global_step‚749 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13737016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13354457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474360543744 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 315313152 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32823380897792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28198904406624 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33846408 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30995926133760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33821534 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13737014 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8121022 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29521547587072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474127628800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474127628800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13353759 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29521547587072 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚ 889.549018065 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309035008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5574211 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33364275499520 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1565938064916 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21420626 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8650318 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30037874881040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚799 _step‚61 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14632491 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1570379309568 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33016942709248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21395780 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34964190096896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14632489 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14224531 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14632491 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5573511 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21420626 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31446563399680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816äÿˆ¡ ç¿O M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36053117 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33017193613824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1570379309568 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309035008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5574211 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33364275499520 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1565938064916 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21420626 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8650318 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30037874881040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚799 _step‚61 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14632491 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1570379309568 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33016942709248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21395780 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34964190096896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14632489 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14224531 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14632491 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5573511 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21420626 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31446563399680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816=ˆ2 Z0è+ _wandbruntime‚948  _runtime‚948yˆmË&éÅM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1572576250368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36073236 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1568125527646 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14241795 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8661122 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30075422277824 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1572576250368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1601629089792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14650766 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21422472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8661120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33405983409664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14650764 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚799 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31485867439616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14650764 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21447344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31485867439616 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35007893386752 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31643547805470 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30075405238464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36098110 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31485849436672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5581355 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31485849436672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885299646800003e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33058443689984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14650766 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5580675 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36098110 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 5602812299¹NÏ&êÉM R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31643301792366 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1572343335424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14241795 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8661122 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30075422277824 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1572576250368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1601629089792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14650766 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21422472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8661120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33405983409664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14650764 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚799 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31485867439616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14650764 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21447344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31485867439616 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35007893386752 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31643547805470 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30075405238464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36098110 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31485849436672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5581355 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31485849436672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885299646800003e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33058443689984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14650766 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5580675 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36098110 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1572576250368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36073236 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1568125527646 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E'/+á0ë+ _wandbruntime‚948  _runtime‚948¿º½VìÐ 2loss/SRqqt1/assignment_loss‚3.0061934730585884 /loss/FRt3/detection_loss‚1.6186849498885802 /loss/FBt3/detection_loss‚1.3828536192586063 /loss/FRt1/assignment_loss‚6.155090727124896 2loss/SRqqt3/assignment_loss‚2.5694450084120035 2loss/SRqqt4/assignment_loss‚2.9620829502741497 1loss/FBt4/assignment_loss‚0.27191380012372346 /loss/FBt4/detection_loss‚1.8659772343676666 % _timestamp‚1.7885299646828034e+09 0loss/FBt2/assignment_loss‚0.3197957571956067 /loss/FRt1/detection_loss‚2.7595341835703167 /loss/FRt2/detection_loss‚3.6924611462487116 1loss/SRqqt2/detection_loss‚2.6988071743398905 .loss/FBt1/detection_loss‚3.942898854613304 .loss/FBt2/detection_loss‚4.054318949580193 trainer/global_step‚799 /loss/FRt3/assignment_loss‚4.366264493986108 0loss/FBt1/assignment_loss‚0.3901589251993869 /loss/FRt4/detection_loss‚2.5635850425987017 %loss/total_loss‚67.29998029836074 _step‚63 /loss/FRt4/assignment_loss‚6.065743616649082 2loss/SRqqt2/assignment_loss‚2.8854218404740095 0loss/FBt3/assignment_loss‚0.4654810647869504 1loss/SRqqt1/detection_loss‚2.6816728378043457 1loss/SRqqt3/detection_loss‚0.8740185289061628 0loss/SRqqt4/detection_loss‚3.073159380753835 _runtime‚ 948.836136567 /loss/FRt2/assignment_loss‚6.634416739145915 epoch‚0Âm¸ËZíÔ 0loss/FBt2/assignment_loss‚0.3197957571956067 /loss/FRt1/detection_loss‚2.7595341835703167 /loss/FRt2/detection_loss‚3.6924611462487116 1loss/SRqqt2/detection_loss‚2.6988071743398905 .loss/FBt1/detection_loss‚3.942898854613304 .loss/FBt2/detection_loss‚4.054318949580193 trainer/global_step‚799 /loss/FRt3/assignment_loss‚4.366264493986108 0loss/FBt1/assignment_loss‚0.3901589251993869 /loss/FRt4/detection_loss‚2.5635850425987017 %loss/total_loss‚67.29998029836074 _step‚63 /loss/FRt4/assignment_loss‚6.065743616649082 2loss/SRqqt2/assignment_loss‚2.8854218404740095 0loss/FBt3/assignment_loss‚0.4654810647869504 1loss/SRqqt1/detection_loss‚2.6816728378043457 1loss/SRqqt3/detection_loss‚0.8740185289061628 0loss/SRqqt4/detection_loss‚3.073159380753835 _runtime‚ 948.836136567 /loss/FRt2/assignment_loss‚6.634416739145915 epoch‚0 2loss/SRqqt1/assignment_loss‚3.0061934730585884 /loss/FRt3/detection_loss‚1.6186849498885802 /loss/FBt3/detection_loss‚1.3828536192586063 /loss/FRt1/assignment_loss‚6.155090727124896 2loss/SRqqt3/assignment_loss‚2.5694450084120035 2loss/SRqqt4/assignment_loss‚2.9620829502741497 1loss/FBt4/assignment_loss‚0.27191380012372346 /loss/FBt4/detection_loss‚1.8659772343676666 % _timestamp‚1.7885299646828034e+09?1Ñï‘î‚(:† ·šëÔÒ²³— gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7424 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ¯,sï‚(:è ·šëÔܵê™ disk.sda2.in‚0 network.sent‚ 1.107515e+06# disk.dm-0.out‚18020.81591796875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1328125 disk.sda.out‚ 30.79296875 disk.nvme0n1.in‚ 2280.7734375 disk.sda3.out‚ 30.79296875) proc.memory.percent‚17.90628433227539% disk./.usageGB‚1177.0386848449707 disk.dm-0.in‚ 2280.7734375 disk.sda.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚18020.81591796875 disk.sda3.in‚0$ memory_percent‚85.59699058532715( proc.memory.availableMB‚ 2359.7890625 network.recv‚386110! proc.memory.rssMB‚ 2933.765625 cpu‚25.12701051588186 proc.cpu.threads‚17) disk./.usagePercent‚78.99205374118351" disk.nvme0c0n1.in‚ 2280.7734375( disk.nvme0c0n1.out‚18020.81591796875 disk.dm-1.in‚0 disk.dm-1.out‚ 30.79296875 disk.sda1.in‚0 disk.sda1.out‚0õä‘ð‚(:† ÆšëÔ ­¾– gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8716 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï4 ×rñ‚(:ç ÆšëÔëš™ disk.sda.out‚ 31.19140625 network.sent‚ 1.142952e+06 disk.sda2.out‚0' disk.nvme0c0n1.out‚18343.2216796875 disk.drbd1260.in‚0 cpu‚25.126395024390924 proc.cpu.threads‚17) disk./.usagePercent‚78.99205246114583 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 2280.7734375 disk.dm-1.in‚0 disk.sda1.in‚0$ memory_percent‚85.59660911560059! proc.memory.rssMB‚ 2933.765625 disk.nvme0n1.in‚ 2280.7734375 disk.sda3.out‚ 31.19140625" disk.dm-0.out‚18339.3466796875 disk.dm-1.out‚ 31.19140625 disk.sda1.out‚0 disk.drbd1260.out‚ 0.13671875( proc.memory.availableMB‚ 2359.8515625) proc.memory.percent‚17.90628433227539 network.recv‚390457% disk./.usageGB‚1177.0386657714844% disk.nvme0n1.out‚18343.2216796875" disk.nvme0c0n1.in‚ 2280.7734375 disk.sda.in‚0£°Ræ‘ò‚(:† ÕšëÔ’Œ‰— gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1616 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£äðø?ó‚(:ç ÕšëÔŒ«™š) proc.memory.percent‚17.906284332lq,327539 network.sent‚ 1.151269e+06 network.recv‚395442 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.13671875$ memory_percent‚85.59815883636475 cpu‚25.12620207016557 proc.cpu.threads‚17% disk./.usageGB‚1177.0386085510254" disk.nvme0c0n1.in‚ 2280.7734375' disk.nvme0c0n1.out‚18563.0458984375 disk.dm-0.in‚ 2280.7734375 disk.dm-1.out‚ 31.59765625 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 2280.7734375" disk.dm-0.out‚18563.0458984375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 31.59765625 disk.sda2.out‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 2359.59765625! proc.memory.rssMB‚ 2933.765625) disk./.usagePercent‚78.99204862103282 disk.sda1.in‚0% disk.nvme0n1.out‚18563.0458984375 disk.sda3.out‚ 31.59765625kÝGó2ô- _wandbruntime‚1004  _runtime‚1004¯ž“›Åõ¿ trainer/global_step‚849 % _timestamp‚1.7885300208322089e+09 _runtime‚1004.985783469 _step‚64 & lr-AdamW/pg1‚0.0005734516173134291 & lr-AdamW/pg2‚0.0005734516173134291-‹ƒëÉöà & lr-AdamW/pg1‚0.0005734516173134291 & lr-AdamW/pg2‚0.0005734516173134291 trainer/global_step‚849 % _timestamp‚1.7885300208322089e+09 _runtime‚1004.985783469 _step‚64@ªÞ¹2÷- _wandbruntime‚1004  _runtime‚1004É”e2Â'ø¼O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38279971 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35079460265472 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35079711170048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1668595016192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22733732 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31914409791600 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1699857695744 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15546241 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9190418 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5920655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33578268857664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31914392752240 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1004.987369489 _step‚65 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37148048699904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33410865249280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1663876105424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35079711170048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582]&¤Æ'ùÀO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31914392752240 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1004.987369489 _step‚65 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37148048699904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33410865249280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1663876105424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35079711170048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38279971 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35079460265472 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35079711170048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1668595016192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22733732 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31914409791600 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1699857695744 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15546241 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9190418 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5920655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33578268857664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15564516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31951923109664 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 300633088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1666063568154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33450169289216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38349812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5927452 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5928133 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22785296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15564514 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33450151286272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15564514 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22760424 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35120710328320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1670559042048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚66 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 _runtime‚1006.068348374 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1670791956992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35490181912064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1670559042048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22760424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15128672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33618003717178 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 E6DeviceStatsMonitor.on_train_batch_end/seg46\6ment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38324938 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33617757704074ïsë/Ï&þÉM BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5928133 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22785296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15564514 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33450151286272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15564514 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22760424 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35120710328320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1670559042048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚66 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 _runtime‚1006.068348374 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1670791956992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35490181912064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1670559042048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22760424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15128672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33618003717178 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38324938 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33617757704074 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15564516 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚849 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37192035900416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1665834594410 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15564516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31951923109664 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 300633088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1666063568154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33450169289216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38349812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5927452 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 Fol…‚(:á ‚›ëÔÁÊô™ disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 2359.7421875) proc.memory.percent‚17.90628433227539 cpu‚25.125115543754987 disk.sda.out‚ 33.0546875 disk.sda3.out‚ 33.0546875 disk.dm-1.out‚ 33.0546875 disk.drbd1260.out‚ 0.14453125! proc.memory.rssMB‚ 2933.765625% disk./.usageGB‚1175.4358596801758! disk.nvme0n1.in‚ 2405.00390625# disk.nvme0n1.out‚19710.46484375 disk.sda3.in‚0% disk.nvme0c0n1.out‚19710.46484375$ memory_percent‚85.59727668762207 proc.cpu.threads‚17 network.sent‚ 1.232616e+06 network.recv‚408368# disk.nvme0c0n1.in‚ 2405.00390625 disk.dm-0.in‚ 2405.00390625 disk.dm-0.out‚19710.46484375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.88448679951424 disk.sda.in‚0ÏYâ·›†‚(: ‘›ëÔö”ß– gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.912' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.982400000000002 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨fh‡d‡‚(:Ù ‘›ëÔöý…™% disk.nvme0c0n1.out‚19968.58203125! disk.nvme0n1.in‚ 2405.00390625# disk.nvme0n1.out‚19968.58203125# disk.nvme0c0n1.in‚ 2405.00390625 disk.sda1.out‚0 disk.drbd1260.out‚ 0.14453125$ memory_percent‚85.60025691986084 network.sent‚ 1.239035e+06 disk.sda.out‚33.5625 disk.sda2.in‚0 disk.dm-0.in‚ 2405.00390625) proc.memory.percent‚17.90628433227539 cpu‚25.124809846685462 network.recv‚413315) disk./.usagePercent‚78.88448295940121% disk./.usageGB‚1175.4358024597168 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.out‚19968.58203125 disk.dm-1.in‚0 disk.dm-1.out‚33.5625 disk.sda3.in‚0 disk.sda3.out‚33.5625 disk.sda1.in‚0 disk.drbd1260.in‚0) proc.memory.availableMB‚ 2359.25390625! proc.memory.rssMB‚ 2933.765625 proc.cpu.threads‚17u¸¹2ˆ- _wandbruntime‚1061  _runtime‚1061Ê”¢Bɽ trainer/global_step‚899 $ _timestamp‚1.788530077353739e+09 _runtime‚ 1061.50732582 _step‚68 & lr-AdamW/pg1‚0.0006072237973672235 & lr-AdamW/pg2‚0.0006072237973672235 ‘­àÇŠÁ & lr-AdamW/pg2‚0.0006072237973672235 trainer/global_step‚899 $ _timestamp‚1.788530077353739e+09 _runtime‚ 1061.50732582 _step‚68 & lr-AdamW/pg1‚0.0006072237973672235Ds!ó2‹- _wandbruntime‚1061  _runtime‚1061†«ÅÂ'Œ¼O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35552970782268 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16459991 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚69 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9730520 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33790910623440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39332499005952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37142228726272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚1061.508677428 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24071684 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1799519563776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16459989 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37532672504320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24096530 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1766810722816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35375185101824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1766810722816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37532696444416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1761814145932 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24071684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1767043624448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39332192068096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40531673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16459991 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35375167098880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35552724769372 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37141977821696 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33790927662800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15995494 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚899 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35375185101824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24096530 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6264974 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309i•v56ÀO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40556521 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16459989 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35375167098880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15994810 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9730518 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37142228726272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚1061.508677428 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24071684 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1799519563776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16459989 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37532672504320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24096530 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1766810722816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35375185101824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1766810722816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37532696444416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1761814145932 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24071684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1767043624448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39332192068096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40531673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16459991 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35375167098880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.lagU'rge_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35552724769372 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37141977821696 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33790927662800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15995494 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚899 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35375185101824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24096530 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6264974 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35552970782268 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16459991 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚69 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9730520 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33790910623440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39332499005952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24098376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1769007663616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35414453135872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16478266 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6271774 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37574380414464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16478264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚35414453135872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24098376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9741322 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33828458020224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40601514 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1768774748672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚1062.476661296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40576640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1769007663616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16478266 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37183478802432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35592213615782 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40576640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16012421 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35414471138816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 $ _timestamp‚1.788530078322468e+09 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24123248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39376188006912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1763772634918 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24123248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39376188006912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1763772634918 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24098376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1769007663616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35414453135872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16478266 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6271774 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37574380414464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16478264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚35414453135872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24098376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9741322 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33828458020224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40601514 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1768774748672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚1062.476661296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40576640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1769007663616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16478266 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37183478802432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35592213615782 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40576640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16012421 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35414471138816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072F÷92‘- _wandbruntime‚1062  _runtime‚1062õ³@žV’Ð 2loss/SRqqt4/assignment_loss‚2.7043312901542302 /loss/FRt2/detection_loss‚2.9681707909612944 0loss/SRqqt1/detection_loss‚3.139639669940585 1loss/SRqqt2/detection_loss‚2.6987164826954113 _runtime‚1062.478463604 1loss/SRqqt2/assignment_loss‚2.686274207690183 1loss/SRqqt3/detection_loss‚0.9127675768190114 1loss/SRqqt4/detection_loss‚3.4805445089226676 epoch‚0 0loss/FRt1/assignment_loss‚6.2574801254272465 0loss/FRt3/assignment_loss‚3.3014278258245016 0loss/FBt3/assignment_loss‚0.6048469766950159 /loss/FRt3/detection_loss‚1.5659895634528287 /loss/FRt4/detection_loss‚3.5342223167419435 /loss/FBt3/detection_loss‚1.5088335921523373 /loss/FBt4/detection_loss‚1.8353493275967512 trainer/global_step‚899 2loss/SRqqt3/assignment_loss‚2.1854178853506268 0loss/FBt4/assignment_loss‚0.3860616605096518 .loss/FRt1/detection_loss‚2.753743119239807 # _timestamp‚1.78853007832504e+09 /loss/FRt4/assignment_loss‚7.265565752983093 _step‚71 0loss/FBt1/assignment_loss‚0.4874738845943424 1loss/SRqqt1/assignment_loss‚2.621221370640255 1loss/FBt2/assignment_loss‚0.34892123933340713 /loss/FBt1/detection_loss‚3.2288743486771216 /loss/FBt2/detection_loss‚3.2497777427945818 %loss/total_loss‚65.36207798256152 /loss/FRt2/assignment_loss‚5.636426723364628ö+P‘Z“Ô 1loss/SRqqt2/detection_loss‚2.6987164826954113 _runtime‚1062.478463604 1loss/SRqqt2/assignment_loss‚2.686274207690183 1loss/SRqqt3/detection_loss‚0.9127675768190114 1loss/SRqqt4/detection_loss‚3.4805445089226676 epoch‚0 0loss/FRt1/assignment_loss‚6.2574801254272465 0loss/FRt3/assignment_loss‚3.3014278258245016 0loss/FBt3/assignment_loss‚0.6048469766950159 /loss/FRt3/detection_loss‚1.5659895634528287 /loss/FRt4/detection_loss‚3.5342223167419435 /loss/FBt3/detection_loss‚1.5088335921523373 /loss/FBt4/detection_loss‚1.8353493275967512 trainer/global_step‚899 2loss/SRqqt3/assignment_loss‚2.1854178853506268 0loss/FBt4/assignment_loss‚0.3860616605096518 .loss/FRt1/detection_loss‚2.753743119239807 # _timestamp‚1.78853007832504e+09 /loss/FRt4/assignment_loss‚7.265565752983093 _step‚71 0loss/FBt1/assignment_loss‚0.4874738845943424 1loss/SRqqt1/assignment_loss‚2.621221370640255 1loss/FBt2/assignment_loss‚0.34892123933340713 /loss/FBt1/detection_loss‚3.2288743486771216 /loss/FBt2/detection_loss‚3.2497777427945818 %loss/total_loss‚65.36207798256152 /loss/FRt2/assignment_loss‚5.636426723364628 2loss/SRqqt4/assignment_loss‚2.7043312901542302 /loss/FRt2/detection_loss‚2.9681707909612944 0loss/SRqqt1/detection_loss‚3.139639669940585GÆ-Íxœ”‚(:‘  ›ëÔƒÈê˜ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚131.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.379199999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÛut•‚(:é  ›ëԵŽœ disk.dm-0.in‚ 2536.77734375 disk.dm-1.out‚ 33.95703125 disk.sda.out‚ 33.95703125 disk.sda3.in‚0 disk.sda3.out‚ 33.95703125' disk.nvme0c0n1.out‚20275.8173828125) proc.memory.percent‚17.90628433227539 cpu‚25.124309652848666 network.sent‚ 1.254822e+06 network.recv‚416052) disk./.usagePercent‚78.88468827744369% disk./.usageGB‚1175.4388618469238 disk.sda1.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 2536.77734375% disk.nvme0n1.out‚20275.8173828125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.1484375 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 2536.77734375$ memory_percent‚85.60070991516113( proc.memory.availableMB‚ 2359.1796875" disk.dm-0.out‚20275.8173828125 disk.sda1.out‚0! proc.memory.rssMB‚ 2933.765625 proc.cpu.threads‚17ödÑ£‘–‚(:† ¯›ëÔ’õ¨— gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.469' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6938 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðkñ.t—‚(:é ¯›ëÔ¦ÐÏš% disk.nvme0n1.out‚20603.7431640625$ memory_percent‚85.60070991516113 disk.sda1.in‚0 disk.sda3.in‚0) proc.memory.percent‚17.90628433227539 proc.cpu.threads‚17 network.recv‚421962) disk./.usagePercent‚78.88311331909304 disk.sda1.out‚0 disk.sda.out‚ 34.41015625 disk.sda2.out‚0( proc.memory.availableMB‚ 2359.1796875! proc.memory.rssMB‚ 2933.765625 cpu‚25.124293881028567 network.sent‚ 1.292823e+06% disk./.usageGB‚1175.4153938293457# disk.nvme0c0n1.in‚ 2536.77734375 disk.dm-1.out‚ 34.41015625 disk.drbd1260.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 2536.77734375 disk.sda3.out‚ 34.41015625' disk.nvme0c0n1.out‚20603.7431640625 disk.dm-0.in‚ 2536.77734375" disk.dm-0.out‚20603.7431640625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.1484375 disk.sda.in‚0ù\‘˜‚(:† ¾›ëÔŹ— gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.791' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3582 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x@w™‚(:ì ¾›ëÔªŸš disk.sda.out‚ 35.08203125 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2359.67578125! proc.memory.rssMB‚ 2933.765625& disk.nvme0n1.out‚20886.18212890625 disk.dm-1.in‚0 disk.dm-1.out‚ 35.08203125 disk.drbd1260.out‚ 0.15234375) proc.memory.percent‚17.90628433227539 cpu‚25.123767764298805 proc.cpu.threads‚17! disk.nvme0n1.in‚ 2536.77734375 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 2536.77734375( disk.nvme0c0n1.out‚20886.18212890625 disk.dm-0.in‚ 2536.77734375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 network.sent‚ 1.295745e+06% disk./.usageGB‚1175.4388427734375 disk.sda3.out‚ 35.08203125# disk.dm-0.out‚20886.18212890625$ memory_percent‚85.59768199920654 network.recv‚425150' disk./.usagePercent‚78.884686997406 disk.sda1.out‚0íS7"œš‚(:‘ Í›ëԇ܂— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚138.022' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.604399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ·_‰ ›‚(:ê Í›ëÔŒ™°™ disk.dm-0.in‚ 2536.77734375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 35.48828125 disk.sda3.out‚ 35.48828125 disk.drbd1260.out‚ 0.15234375! disk.nvme0n1.in‚ 2536.77734375& disk.nvme0n1.out‚21142.00244140625! proc.memory.rssMB‚ 2933.765625) proc.memory.percent‚17.90628433227539 network.recv‚430105# disk.dm-0.out‚21142.00244140625 disk.dm-1.out‚ 35.48828125 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 2359.671875) disk./.usagePercent‚78.87697221036167# disk.nvme0c0n1.in‚ 2536.77734375 disk.drbd1260.in‚0$ memo&¡»Õry_percent‚85.59770584106445 cpu‚25.12349631650425 proc.cpu.threads‚17 network.sent‚ 1.302059e+06$ disk./.usageGB‚1175.323886871338 disk.sda3.in‚0( disk.nvme0c0n1.out‚21142.00244140625}9ø,2œ- _wandbruntime‚1117  _runtime‚1117»ìò?Å¿ & lr-AdamW/pg1‚0.0006409959774210178 & lr-AdamW/pg2‚0.0006409959774210178 trainer/global_step‚949 % _timestamp‚1.7885301329826574e+09 _runtime‚1117.136225195 _step‚72ºº=¸ÉžÃ % _timestamp‚1.7885301329826574e+09 _runtime‚1117.136225195 _step‚72 & lr-AdamW/pg1‚0.0006409959774210178 & lr-AdamW/pg2‚0.0006409959774210178 trainer/global_step‚949H Naf2Ÿ- _wandbruntime‚1117  _runtime‚1117ÔI ;Ä' ¾O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39204746282496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39204495377920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16882940 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37339468948480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1865259331072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37339486951424 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41516755154432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39616894946816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1865026429440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1899564626944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚73 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10270620 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1859752186440 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1117.137749972 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17373739 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35667428494640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39616871006720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚949 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41516755154432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39616894946816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1865026429440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1899564626944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚73 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10270620 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1859752186440 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1117.137749972 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17373739 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35667428494640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39616871006720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚949 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39204746282496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39204495377920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16882940 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37339468948480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1865259331072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37339486951424 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42853216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6618907 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42828342 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17392016 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37378754985472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 298535936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39245745440768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39245996358656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39658578916864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1867223370240 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚949 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6618221 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37566669527490 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35704975891424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17392016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35704958852064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861939649170 _runtime‚1118.285764622 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16899641 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885301341313066e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 274595840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16900329 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37378772988416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37566915540594 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37378754985472 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39245996358656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17392014 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496§Ïpg¤ÌM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17392014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25436328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25461200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41560421529088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37378772988416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1867223370240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37378754985472 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39245996358656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6:,_…k 458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17392014 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42853216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6618907 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42828342 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17392016 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37378754985472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 298535936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39245745440768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39245996358656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39658578916864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1867223370240 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚949 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6618221 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37566669527490 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35704975891424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17392016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35704958852064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861939649170 _runtime‚1118.285764622 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16899641 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885301341313066e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 274595840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16900329 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37378772988416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37566915540594 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41267012934144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1963475037696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7885301910729566e+09 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1957690226948 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1963475037696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26747588 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39303770798080 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45059925 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37543946365840 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18287491 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39303788801024 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41701093449216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39501636592788 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10810718 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41267263838720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26747588 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18287489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41267012934144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1963242136064 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1999683520000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26772434 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6959637 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39303788801024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18287491 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚1175.226827365’m‰ŽµÂO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41701093449216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39501636592788 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10810718 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41267263838720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26747588 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18287489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41267012934144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1963242136064 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1999683520000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26772434 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6959637 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39303788801024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18287491 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚1175.226827365 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17770357 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18287489 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39501882605684 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41701069509120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37543963405200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1957919200484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41267263838720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45035077 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1963242136064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6958936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39303770798080 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved@dEØ:_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1999974906368 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41267012934144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1963475037696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7885301910729566e+09 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1957690226948 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1963475037696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26747588 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39303770798080 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45059925 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37543946365840 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18287491 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39303788801024 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0M/×2¶- _wandbruntime‚1176  _runtime‚1176Ó$VÍ&·ÇM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26774280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45080044 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1965439076864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1959877689678 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 1176.35076799 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6966839 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6966165 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37581493762624 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26774280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41308513914880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39343074838016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39343056835072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1965206161920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18305766 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26799152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2001677341184 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43744741939200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41308262996992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18305766 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37581476723264 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1959648715934 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45104918 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39343074838016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41742777419264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1965439076864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41742753479168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚999 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43744430820352 % _timestamp‚1.7885301921961944e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1965206161920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18305764 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39541371452302 _step‚78 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17788361 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10821520 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26799152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45104918 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18305764 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39343056835072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17787685 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39541125439198 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26799152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45104918 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18305764 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39343056835072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17787685 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39541125439198 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39343074838016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39343056835072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1965206161920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18305766 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26799152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2001677341184 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43744741939200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41308262996992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18305766 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37581476723264 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1959648715934 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45104918 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39343074838016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41742777419264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1965439076864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41742753479168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚999 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43744430820352 % _timestamp‚1.7885301921961944e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1965206161920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18305764 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39541371452302 _step‚78NÊ·Z«2¹- _wandbruntime‚1176  _runtime‚1176P¹"UºÏ _step‚79 2loss/SRqqt1/assignment_loss‚2.6633318174969065 .loss/FBt2/detection_loss‚3.354864038527012 /loss/FRt3/assignment_loss‚2.562344478313313 /loss/FBt4/detection_loss‚1.2982472904040006 trainer/global_step‚999 /loss/FRt3/detection_loss‚1.5348888659026734 /loss/FRt4/detection_loss‚3.4740445415178933 /loss/FRt2/assignment_loss‚5.807549728287591 2loss/SRqqt2/assignment_loss‚2.3823143352161753 2loss/SRqqt4/assignment_loss‚3.0146052136140713 1loss/FBt1/assignment_loss‚0.31416772267084525 1loss/SRqqt1/detection_loss‚3.2028150260448456 /loss/FRt1/assignment_loss‚5.952679384322393 .loss/FRt4/assignment_loss‚6.60194460550944 1loss/SRqqt2/detection_loss‚2.8703155969128464 /loss/FBt1/detection_loss‚3.7602661103010178 %loss/total_loss‚64.10704750170613 epoch‚0 % _timestamp‚1.7885301921990066e+09 0loss/FBt3/assignment_loss‚0.5624887035802894 0loss/FBt4/assignment_loss‚0.2895269280356249 -loss/FRt2/detection_loss‚2.53895922170745 /loss/FBt3/detection_loss‚2.3793154386458575 1loss/SRqqt3/detection_loss‚0.8281758294535287 0loss/SRqqt4/detection_loss‚3.402758037342745 _runtime‚1176.352408564 2loss/SRqqt3/assignment_loss‚2.1164924495893978 0loss/FBt2/assignment_loss‚0.6184302336146175 /loss/FRt1/detection_loss‚2.5765219046956016‹[ÐMY»Ó /loss/FRt3/detection_loss‚1.5348888659026734 /loss/FRt4/detection_loss‚3.4740445415178933 /loss/FRt2/assignment_loss‚5.807549728287591 2loss/SRqqt2/assignment_loss‚2.3823143352161753 2loss/SRqqt4/assignment_loss‚3.0146052136140713 1loss/FBt1/assignment_loss‚0.31416772267084525 1loss/SRqqt1/detection_loss‚3.2028150260448456 /loss/FRt1/assignment_loss‚5.952679384322393 .loss/FRt4/assignment_loss‚6.60194460550944 1loss/SRqqt2/detection_loss‚2.8703155969128464 /loss/FBt1/detection_loss‚3.7602661103010178 %loss/total_loss‚64.10704750170613 epoch‚0 % _timestamp‚1.7885301921990066e+09 0loss/FBt3/assignment_loss‚0.5624887035802894 0loss/FBt4/assignment_loss‚0.2895269280356249 -loss/FRt2/detection_loss‚2.53895922170745 /loss/FBt3/detection_loss‚2.3793154386458575 1loss/SRqqt3/detection_loss‚0.8281758294535287 0loss/SRqqt4/detection_loss‚3.402758037342745 _runtime‚1176.352408564 2loss/SRqqt3/assignment_loss‚2.1164924495893978 0loss/FBt2/assignment_loss‚0.6184302336146175 /loss/FRt1/detection_loss‚2.5765219046956016 _step‚79 2loss/SRqqt1/assignment_loss‚2.6633318174969065 .loss/FBt2/detection_loss‚3.354864038527012 /loss/FRt3/assignment_loss‚2.562344478313313 /loss/FBt4/detection_loss‚1.2982472904040006 trainer/global_step‚999O„‘Gßœ¼‚(:‘ ˜œëÔË Ä— gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.656' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.131200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÝv†r½‚(:ç ˜œëÔºÕ×™ disk.sda3.in‚0# disk.dm-0.out‚22454.27783203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.1640625! proc.memory.rssMB‚ 2933.765625 network.sent‚ 1.373039e+06 network.recv‚450765" disk.nvme0c0n1.in‚ 2716.4296875( disk.nvme0c0n1.out‚22454.27783203125 disk.dm-1.out‚ 37.7734375 disk.drbd1260.in‚0$ memory_percent‚85.60168743133545) proc.memory.percent‚17.90628433227539% disk./.usageGB‚1175.3007164001465 disk.sda.in‚0 disk.sda.out‚ 37.7734375 disk.sda2.in‚0& disk.nvme0n1.out‚22454.27783203125 disk.sda3.out‚ 37.7734375 disk.nvme0n1.in‚ 2716.4296875 disk.dm-0.in‚ 2716.4296875 disk.dm-1.in‚0) proc.memory.availableMB‚ 2359.01953125 cpu‚25.121836683990463 proc.cpu.threads‚17) disk./.usagePercent‚78.87541722059869 disk.sda2.out‚0e.Ø‘¾‚(:† §œëÔÁ”»— gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.722' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7444 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fh\¿‚(:Ñ §œëԳ܈š disk.sda.in‚0 disk.sda.out‚38.25 disk.nvme0n1.in‚ 2889.015625% disk./.usageGB‚1175.3237762451172' disk.nvme0c0n1.out‚22734.2978515625" disk.dm-0.out‚22734.2978515625 disk.dm-1.out‚38.25 disk.drbd1260.out‚ 0.1640625! proc.memory.rssMB‚ 2933.765625 cpu‚25.12175761065876 network.sent‚ 1.424767e+06 network.recv‚456345 disk.sda1.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚22734.2978515625 disk.sda3.out‚38.25) proc.memory.availableMB‚ 2358.77734375) proc.memory.percent‚17.90628433227539) disk./.usagePercent‚78.87696478614319 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚85.60316562652588 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 2889.015625 disk.dm-0.in‚ 2889.015625 disk.drbd1260.in‚0wª"œÀ‚(:‘ ¶œëÔ·¸Å– gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚138.022' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.604399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åžYiÁ‚(:Þ ¶œëÔž˜ç˜ network.sent‚ 1.427689e+06) disk./.usagePercent‚78.87696325009797 disk.drbd1260.out‚ 0.16796875 disk.sda2.out‚0 disk.nvme0n1.in‚ 2889.015625! disk.nvme0c0n1.in‚ 2889.015625! proc.memory.rssMB‚ 2933.765625 cpu‚25.121348808100002 disk.dm-0.out‚22804.10546875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 38.69921875$ memory_percent‚85.60166358947754 disk.dm-0.in‚ 2889.015625 disk.sda1.out‚0# disk.nvme0n1.out‚22804.10546875% disk.nvme0c0n1.out‚22804.10546875( proc.memory.availableMB‚ 2359.0234375) proc.memory.percent‚17.90628433227539 proc.cpu.threads‚17 network.recv‚459475% disk./.usageGB‚1175.3237533569336 disk.dm-1.out‚ 38.69921875 disk.sda.out‚ 38.69921875 disk.sda3.in‚0 z’‚(:„ ÅœëÔÅ£— gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.345' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.669 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[AˆâcÂ(:Ø ÅœëÔ£¨“š disk.sda.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 2889.015625 disk.sda3.out‚ 39.1796875 network.sent‚ 1.434145e+06) disk./.usagePercent‚78.83938160007904! disk.nvme0c0n1.in‚ 2889.015625 disk.dm-0.in‚ 2889.015625 disk.drbd1260.out‚ 0.16796875! proc.memory.rssMB‚ 2933.765625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 39.1796875$ memory_percent‚85.60163974761963) proc.memory.availableMB‚ 2359.02734375) proc.memory.percent‚17.90628433227539$ disk./.usageGB‚1174.763759613037$ disk.nvme0c0n1.out‚ 22939.7890625 disk.dm-1.out‚ 39.1796875 disk.sda2.in‚0" disk.nvme0n1.out‚ 22939.7890625 disk.sda3.in‚0 cpu‚25.121039870620255 proc.cpu.threads‚17 network.recv‚464266 disk.dm-0.out‚ 22939.7890625¿Ÿ Á2Ä- _wandbruntime‚1234  _runtime‚1234õMùÆÅÀ & lr-AdamW/pg1‚0.0007085403375286066 & lr-AdamW/pg2‚0.0007085403375286066 trainer/global_step‚1049 % _timestamp‚1.7885302501651528e+09 _runtime‚1234.318584941 _step‚80åy¶#®ÆÄ trainer/global_step‚1049 % _timestamp‚1.7885302501651528e+09 _runtime‚1234.318584941 _step‚80 & lr-AdamW/pg1‚0.0007085403375286066 & lr-AdamW/pg2l¤èˆ‚0.0007085403375286066PËè9‹2Ç- _wandbruntime‚1234  _runtime‚1234I"0ðÅ'È¿O UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19201239 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18658535 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41268090650624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39420481276400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2099577829376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2055628267456 _step‚81 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41268090650624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2099867118592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚41268072647680 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19201241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚43329781394944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41476092504496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885302501660786e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41268072647680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45885159070208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43785268011520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47286779 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2061690744320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28110386 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚43329530490368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41476338517392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11350820 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7308428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2055857240992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚43329781394944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19201241 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47311627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 _runtime‚1234.320012572 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39420464237040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43329530490368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28085540 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2061457842688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28110386 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19201239 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28085540 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚1049 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43329530490368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28085540 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2061457842688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28110386 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19201239 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28085540 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚1049 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19201241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚43329781394944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41476092504496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885302501660786e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41268072647680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45885159070208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43785268011520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47286779Q™ç2Ê- _wandbruntime‚1235  _runtime‚1235û‘=XÏ&ËÉM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7314300 trainer/global_step‚1049 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2057586756442 _runtime‚1235.561079855 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚43371031471104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2101573633024 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45928834636288 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18675920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2057815730186 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45928525614592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚43371031471104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41307358684672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19219514 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚41307358684672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2063654783488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28137104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47331746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39458011633824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7314976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19219516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2063421868544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47356620 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41307376687616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚43370780553216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41307376687616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2063654783488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19219516 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18676598 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39457994594464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11361620 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28137104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11361622 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2101858714624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19219514 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7885302514067752e+09 _step‚82 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47331746 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28112232 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309XxˆôòÌÍM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2063654783488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28137104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47331746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39458011633824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7314976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19219516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMoniÀ¯dÆátor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2063421868544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47356620 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41307376687616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚43370780553216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41307376687616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2063654783488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19219516 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18676598 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39457994594464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11361620 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28137104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11361622 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2101858714624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19219514 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7885302514067752e+09 _step‚82 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47331746 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28112232 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7314300 trainer/global_step‚1049 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2057586756442 _runtime‚1235.561079855 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚43371031471104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2101573633024 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45928834636288 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18675920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2057815730186 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45928525614592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚43371031471104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41307358684672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19219514 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚41307358684672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496RbÖý2Í- _wandbruntime‚1235  _runtime‚1235¬“–9UÎÏ %loss/total_loss‚62.69526783815087 epoch‚0 /loss/FRt1/assignment_loss‚6.140919723008809 /loss/FRt2/assignment_loss‚5.329110041260719 1loss/FBt2/assignment_loss‚0.40450037536033456 1loss/SRqqt1/detection_loss‚2.6813407278060915 _step‚83 /loss/FRt2/detection_loss‚2.9222025014460087 /loss/FBt4/detection_loss‚1.3241254588230793 1loss/FBt3/assignment_loss‚0.44954373111749585 /loss/FRt4/detection_loss‚3.2591505279907813 trainer/global_step‚1049 /loss/FRt4/assignment_loss‚6.768848914366502 2loss/SRqqt1/assignment_loss‚3.0650852793455123 1loss/SRqqt3/detection_loss‚0.7974993970299906 1loss/SRqqt2/detection_loss‚2.5865034370711357 .loss/FBt2/detection_loss‚4.413293381532033 /loss/FRt3/assignment_loss‚2.555159256599917 2loss/SRqqt2/assignment_loss‚2.8396555907798535 1loss/FBt1/assignment_loss‚0.16349970426087815 /loss/FRt3/detection_loss‚1.4004244361773577 2loss/SRqqt4/assignment_loss‚2.4906825125217438 1loss/SRqqt4/detection_loss‚3.1824294894933702 .loss/FBt1/detection_loss‚3.091450647874312 .loss/FRt1/detection_loss‚2.523701771309501 -loss/FBt3/detection_loss‚2.13592418147759 2loss/SRqqt3/assignment_loss‚1.9740341622382402 0loss/FBt4/assignment_loss‚0.1961825892596043 " _timestamp‚1.7885302514089e+09 _runtime‚1235.562346723Ѻ.YÏÓ 1loss/SRqqt3/detection_loss‚0.7974993970299906 1loss/SRqqt2/detection_loss‚2.5865034370711357 .loss/FBt2/detection_loss‚4.413293381532033 /loss/FRt3/assignment_loss‚2.555159256599917 2loss/SRqqt2/assignment_loss‚2.8396555907798535 1loss/FBt1/assignment_loss‚0.16349970426087815 /loss/FRt3/detection_loss‚1.4004244361773577 2loss/SRqqt4/assignment_loss‚2.4906825125217438 1loss/SRqqt4/detection_loss‚3.1824294894933702 .loss/FBt1/detection_loss‚3.091450647874312 .loss/FRt1/detection_loss‚2.523701771309501 -loss/FBt3/detection_loss‚2.13592418147759 2loss/SRqqt3/assignment_loss‚1.9740341622382402 0loss/FBt4/assignment_loss‚0.1961825892596043 " _timestamp‚1.7885302514089e+09 _runtime‚1235.562346723 %loss/total_loss‚62.69526783815087 epoch‚0 /loss/FRt1/assignment_loss‚6.140919723008809 /loss/FRt2/assignment_loss‚5.329110041260719 1loss/FBt2/assignment_loss‚0.40450037536033456 1loss/SRqqt1/detection_loss‚2.6813407278060915 _step‚83 /loss/FRt2/detection_loss‚2.9222025014460087 /loss/FBt4/detection_loss‚1.3241254588230793 1loss/FBt3/assignment_loss‚0.44954373111749585 /loss/FRt4/detection_loss‚3.2591505279907813 trainer/global_step‚1049 /loss/FRt4/assignment_loss‚6.768848914366502 2loss/SRqqt1/assignment_loss‚3.0650852793455123S»©cЂ(:… ÔœëÔêþ– gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚105.487' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0974 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿæ¤mÑ‚(:â ÔœëԌ⪘( disk.nvme0c0n1.out‚23138.83056640625 disk.dm-0.in‚ 2889.015625% disk./.usageGB‚1174.5087623596191 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 1.439011e+06 network.recv‚467382 disk.sda.out‚ 39.6171875 disk.nvme0n1.in‚ 2889.015625& disk.nvme0n1.out‚23138.83056640625! disk.nvme0c0n1.in‚ 2889.015625$ memory_percent‚85.60187816619873! proc.memory.rssMB‚ 2933.765625 disk.dm-1.out‚ 39.6171875 disk.sda1.in‚0 disk.drbd1260.out‚0.171875# disk.dm-0.out‚23138.83056640625) proc.memory.availableMB‚ 2358.98828125) proc.memory.percent‚17.90628433227539 cpu‚25.12067215983812 proc.cpu.threads‚17) disk./.usagePercent‚78.82226852044522 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 39.6171875£ »œÒ‚(:‘ ãœëÔ›˜°– gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.067' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.813399999999998 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡BiÓ‚(:Þ ãœëÔ§†€™ disk.drbd1260.out‚0.171875) proc.memory.percent‚17.90628433227539 disk.dm-0.in‚ 3054.24609375 disk.dm-1.out‚39.9375 disk.sda.out‚39.9375( proc.memory.availableMB‚ 2358.6796875 cpu‚25.120599481781763( disk.nvme0c0n1.out‚23494.16259765625# disk.dm-0.out‚23494.16259765625 disk.dm-1.in‚0 network.sent‚ 1.488769e+06! disk.nvme0n1.in‚ 3054.24609375& disk.nvme0n1.out‚23494.16259765625 disk.sda3.out‚39.9375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 3054.24609375 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚85.60376167297363! proc.memory.rssMB‚ 2933.765625 proc.cpu.threads‚17 network.recv‚472762( disk./.usagePercent‚78.8221046756233% disk./.usageGB‚1174.5063209533691 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0uß«ÊÔ‚(:… òœëÔѤæ– gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0526 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Êæ‰qÕ‚(:æ òœëÔΟ±™ network.recv‚476236( disk.nvme0c0n1.out‚23687.06982421875# disk.dm-0.out‚23687.06982421875 disk.dm-1.out‚ 40.671875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0) disk./.usagePercent‚78.82210134752535! disk.nvme0n1.in‚ 3054.24609375& disk.nvme0n1.out‚23687.06982421875 disk.dm-0.in‚ 3054.24609375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.17578125$ memory_percent‚85.60073375701904 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 40.671875# disk.nvme0c0n1.in‚ 3054.24609375) proc.memory.availableMB‚ 2359.17578125 cpu‚25.12034846657317 proc.cpu.threads‚17% disk./.usageGB‚1174.5062713623047 disk.sda3.in‚0 disk.sda.out‚ 40.671875! proc.memory.rssMB‚ 2933.765625) proc.memory.percent‚17.90628433227539 network.sent‚ 1.49549e+06dààÆ‘Ö‚(:† ëÔÚÿ»™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.171' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6342 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëð~…wׂ(:ì ëÔØÖ‰! proc.memory.rssMB‚ 2933.765625) proc.memory.percent‚17.90628433227539 network.sent‚ 1.501907e+06$ disk./.usageGB‚1174.428653717041 disk.dm-0.in‚ 3054.24609375 disk.drbd1260.out‚ 0.17578125 disk.sda2.out‚0$ memory_percent‚85.60223579406738 proc.cpu.threads‚17# disk.dm-0.out‚23971.01904296875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 41.08203125 cpu‚25.120162573532472) disk./.usagePercent‚78.81689236222617# disk.nvme0c0n1.in‚ 3054.24609375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 41.08203125& disk.nvme0n1.out‚23971.01904296875( proc.memory.availableMB‚ 2358.9296875 network.recv‚481346( disk.nvme0c0n1.out‚23971.01904296875 disk.dm-1.out‚ 41.08203125 disk.sda1.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 3054.24609375B\2È2Ø- _wandbruntime‚1291  _runtime‚1291Θ›ÆÙÀ & lr-AdamW/pg1‚0.0007423125175824012 & lr-AdamW/pg2‚0.0007423125175824012 trainer/global_step‚1099 % _timestamp‚1.7885303072714722e+09 _runtime‚1291.424962563 _step‚84Ø“ÊÚÄ & lr-AdamW/pg1‚0.0007423125175824012 & lr-AdamW/pg2‚0.0007423125175824012 trainer/global_step‚1099 % _timestamp‚1.7885303072714722e+09 _runtime‚1291.424962563 _step‚84T6+«‚2Û- _wandbruntime‚1291  _runtime‚1291 F':Å'Ü¿O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2159906450944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45869490454016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200070349824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11890918 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159673549312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29423492 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2159906450944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7654486 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45869466513920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43232392500224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43232374497280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49538481 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885303072724376e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43232374497280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29448338 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43450548416204 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20114989 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚85 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2199781060608 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45392298951168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45392048046592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43232392500224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20114989 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43450794429100 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45392048046592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49563329 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29448338 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11890920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚1099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45392298951168 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48069247574528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41296999147600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20114991 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚1291.426060923 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2159673549312 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2159906450944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45869490454016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200070349824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11890918 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159673549312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29423492 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2159906450944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7654486 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45869466513920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43232392500224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43232374497280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49538481 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 T)½\ÖÌFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885303072724376e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43232374497280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29448338 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43450548416204 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20114989 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚85 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2199781060608 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45392298951168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45392048046592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43232392500224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20114989 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43450794429100 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45392048046592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49563329 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29448338 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11890920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚1099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45392298951168 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48069247574528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41296999147600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309UÚ® k2Þ- _wandbruntime‚1292  _runtime‚1292M!=wÏ&ßÉM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚1099 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43490037262614 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19563534 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43271660534272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43271678537216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11901722 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2201783197696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43271678537216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49608322 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41334512465664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155753770694 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2161870490112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19562834 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2202068279296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚86 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 % _timestamp‚1.7885303082526739e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43271660534272 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45433549027328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48113242703360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29450184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49583448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2161637575168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20133264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29450184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45911150483968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2155524796950 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45433298109440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2161637575168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1292.406579836 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20133266 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49583448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7661114 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161870490112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29475056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45433549027328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20133266 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29475056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11901720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43490283275718 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29475056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11901720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43490283275718 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19563534 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43271660534272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43271678537216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11901722 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2201783197696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43271678537216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49608322 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41334512465664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155753770694 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2161870490112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19562834 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2202068279296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚86 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 % _timestamp‚1.7885303082526739e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43271660534272 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45433549027328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48113242703360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29450184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49583448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2161637575168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20133264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29450184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45911150483968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2155524796950 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45433298109440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2161637575168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136Vù`œ¢2á- _wandbruntime‚1292  _runtime‚1292Þk£åTâÎ 2loss/SRqqt1/assignment_loss‚2.6573803446122577 2loss/SRqqt2/assignment_loss‚2.2983188159538037 .loss/FRt1/detection_loss‚2.632617525152258 /loss/FBt1/detection_loss‚3.8785556852817535 0loss/FBt2/assignment_loss‚0.2861462118612094 /loss/FRt2/detection_loss‚3.1739720511436462 0loss/SRqqt1/detection_loss‚2.729895200048174 # _timestamp‚1.78853030825508e+09 /loss/FRt1/assignment_loss‚5.434243485734269 1loss/SRqqt3/assignment_loss‚1.871082094715054 0loss/FBt3/assignment_loss‚0.6788154616951942 1loss/SRqqt2/detection_loss‚2.7786191123904604 1loss/SRqqt3/detection_loss‚0.8102997682036314 _runtime‚1292.408406309 2loss/SRqqt4/assignment_loss‚2.9782784948746364 0loss/FRt3/assignment_loss‚2.2374329580817114 /loss/FRt3/detection_loss‚1.5761366295086783 .loss/FRt4/detection_loss‚3.766365948650572 /loss/FBt2/detection_loss‚3.2259693958542566 /loss/FBt3/detection_loss‚1.8760300930589437 /loss/FBt4/detection_loss‚1.7423547270198279 epoch‚0 trainer/global_step‚1099 /loss/FBt4/assignment_loss‚0.288090946209055 $loss/total_loss‚62.7266293343493 /loss/FRt2/assignment_loss‚5.719600610733032 /loss/FRt4/assignment_loss‚6.274814287821452 1loss/FBt1/assignment_loss‚0.38825811445712927 1loss/SRqqt4/detection_loss‚3.4233513712882995 _step‚87E‘©XãÒ 1loss/SRqqt3/assignment_loss‚1.871082094715054 0loss/FBt3/assignment_loss‚0.6788154616951942 1loss/SRqqt2/detection_loss‚2.7786191123904604 1loss/SRqqt3/detection_loss‚0.8102997682036314 _runtime‚1292.408406309 2loss/SRqqt4/assignment_loss‚2.9782784948746364 0loss/FRt3/assignment_loss‚2.2374329580817114 /loss/FRt3/detection_loss‚1.5761366295086783 .loss/FRt4/detection_loss‚3.766365948650572 /loss/FBt2/detection_loss‚3.2259693958542566 /loss/FBt3/detection_loss‚1.8760300930589437 /loss/FBt4/detection_loss‚1.7423547270198279 epoch‚0 trainer/global_step‚1099 /loss/FBt4/assignment_loss‚0.288090946209055 $loss/total_loss‚62.7266293343493 /loss/FRt2/assignment_loss‚5.719600610733032 /loss/FRt4/assignment_loss‚6.274814287821452 1loss/FBt1/assignment_loss‚0.38825811445712927 1loss/SRqqt4/detection_loss‚3.4233513712882995 _step‚87 2loss/SRqqt1/assignment_loss‚2.6573803446122577 2loss/SRqqt2/assignment_loss‚2.2983188159538037 .loss/FRt1/detection_loss‚2.632617525152258 /loss/FBt1/detection_loss‚3.8785556852817535 0loss/FBt2/assignment_loss‚0.2861462118612094 /loss/FRt2/detection_loss‚3.1739720511436462 0loss/SRqqt1/detection_loss‚2.729895200048174 # _timestamp‚1.78853030825508e+09 /loss/FRt1/assignment_loss‚5.434243485734269W5#¯‘ä‚(:† ëÔЈ£— gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.059' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2118 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BÙÒuå‚(:ê ëÔ§–š disk.sda3.out‚ 41.57421875 disk.dm-0.in‚ 3054.24609375) proc.memory.percent‚17.90628433227539 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0# disk.dm-0.out‚24159.32568359375( proc.memory.availableMB‚ 2359.1171875! proc.memory.rssMB‚ 2933.765625 disk.sda.out‚ 41.57421875 disk.sda2.out‚0 network.sent‚ 1.51787e+06) disk./.usagePercent‚78.81690925872343 disk.dm-1.in‚0 disk.dm-1.out‚ 41.57421875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1796875 network.recv‚484862% disk./.usageGB‚1174.4289054870605! disk.nvme0n1.in‚ 3054.24609375# disk.nvme0c0n1.in‚ 3054.24609375( disk.nvme0c0n1.out‚24159.32568359375# memory_percent‚85.6010913848877 cpu‚25.119862218636825 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚24159.32568359375P«²î‘æ‚(:† ŸëÔÆŪ– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5864 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒU²lç‚(:á ŸëԺ) disk./.usagePercent‚78.83934422297904 disk.sda.in‚0 disk.sda2.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 2358.60546875* proc.memory.percent‚17.906761169433594 network.sent‚ 1.555042e+06 disk.sda.out‚42.03125 disk.nvme0n1.in‚ 3173.4453125" disk.nvme0c0n1.in‚ 3173.4453125 disk.dm-0.in‚ 3173.4453125 disk.dm-1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1174.7632026672363 disk.drbd1260.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚24501.86474609375 disk.sda1.in‚0$ memory_percent‚85.60421466827393 proc.memory.rssMB‚ 2933.84375 cpu‚25.119502617621315 disk.drbd1260.out‚ 0.1796875 disk.sda3.in‚0 disk.sda3.out‚42.03125( disk.nvme0c0n1.out‚24501.86474609375# disk.dm-0.out‚24501.86474609375 disk.dm-1.out‚42.03125 network.recv‚491323P“E’‘è‚(:† ®ëÔõêΘ gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8238 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j§gëpé‚(:å ®ëԼ̊ disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.18359375 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 42.73046875 disk.nvme0n1.in‚ 3173.4453125 disk.sda3.out‚ 42.73046875$ memory_percent‚85.60323715209961' proc.memory.availableMB‚ 2358.765625 proc.memory.rssMB‚ 2933.84375* proc.memory.percent‚17.906761169433594 network.recv‚494957) disk./.usagePercent‚78.85281969155885" disk.nvme0c0n1.in‚ 3173.4453125 disk.dm-0.in‚ 3173.4453125" disk.dm-0.out‚24689.2099609375 disk.sda2.in‚0% disk.nvme0n1.out‚24689.2099609375 proc.cpu.threads‚17 network.sent‚ 1.562242e+06' disk.nvme0c0n1.out‚24689.2099609375 disk.sda1.out‚0 cpu‚25.119499413951857$ disk./.usageGB‚1174.963996887207 disk.dm-1.out‚ 42.73046875 disk.sda1.in‚0þ˜N¶ê‚(:‚ ½ëÔÏà´— gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝûÎ4oë‚(:ä ½ëÔ½×Û( disk./.usagePercent‚78.8530091371342 disk.sda.in‚0" disk.dm-0.out‚25204.9677734375 disk.dm-1.in‚0 disk.sda1.in‚0% disk.nvme0n1.out‚25204.9677734375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.18359375) proc.memory.availableMB‚ 2358.76171875 proc.memory.rssMB‚ 2933.84375 cpu‚25.119527437949582" disk.nvme0c0n1.in‚ 3173.4453125 network.recv‚499662% disk./.usageGB‚1174.9668197631836 disk.sda.out‚ 43.1484375 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0$ memory_percent‚85.60326099395752 disk.sda2.out‚0 disk.nvme0n1.in‚ 3173.4453125 disk.dm-0.in‚ 3173.4453125 disk.dm-1.out‚ 43.1484375* proc.memory.percent‚17.906761169433594 proc.cpu.threads‚17 disk.sda3.out‚ 43.1484375' disk.nvme0c0n1.out‚25204.9677734375 network.sent‚ 1.566654e+06=Ä$g2ì- _wandbruntime‚1351  _runtime‚1351-• ïÆíÀ & lr-AdamW/pg1‚0.0007760846976361955 & lr-AdamW/pg2‚0.0007760846976361955 trainer/global_step‚1149 % _timestamp‚1.7885303675196028e+09 _runtime‚1351.673322076 _step‚88¨GÞÊîÄ & lr-AdamW/pg1‚0.0007760846976361955 & lr-AdamW/pg2‚0.0007760846976361955 trainer/global_step‚1149 % _timestamp‚1.7885303675196028e+09 _runtime‚1351.673322076 _step‚88XI³½-2ï- _wandbruntime‚1351  _runtime‚1351n‘Þ¥ð¾O G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47953665016320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2251504348472 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45196676346880 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45425004327912 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 1351.67512311 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21028739 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21028739 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47454816507392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21028741 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20437440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251733322008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8006420 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47953688956416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47454565602816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45196676346880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30761444 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30761444 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45425250340808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43173517018800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47454565602816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885303675210092e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30786290 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 trainer/global_step‚1149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30786290 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2300031541248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45196694349824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12431020 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21028741 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50253720497664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2258122157568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2257889255936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8005719 _step‚89 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45196694349824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896­ È'ñÂO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12431018 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47454565602816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45196676346880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30761444 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30761444 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45425250340808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43173517018800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47454565602816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885303675210092e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30786290 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 trainer/global_step‚1149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30786290 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2300031541248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45196694349824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12431020 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21028741 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50253720497664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2258122157568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2257889255936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8005719 _step‚89 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45196694349824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47953665016320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2251504348472 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45196676346880 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45425004327912 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 1351.67512311 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21028739 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21028739 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47454816507392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21028741 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20437440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251733322008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8006420 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47953688956416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50297082700288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45464739187426 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259853281792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2253691811202 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45235980386816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2260086196736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30813008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47496066583552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8013441 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 289275904 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21047014 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 1353.22508278 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45235962383872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 313216000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45464493174322 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45235980386816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚90 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51835150 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21047016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43211030336864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12441822 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12441820 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47995348986368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51860024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260086196736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47495815665664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30788136 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47496066583552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21047016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20455263 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 $ _timestamp‚1.788530369069955e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30788136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47995372926464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2302022989824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20454572 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50297395916288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2301733713920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259853281792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51835150 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51860024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2253462837458 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689¨o¹T ôËM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2253691811202 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45235980386816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2260086196736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30813008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47496066583552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8013441 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 289275904 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21047014 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 1353.22508278 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45235962383872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 313216000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45464493174322 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45235980386816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.ź٠}peak‚ 336281088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚90 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51835150 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21047016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43211030336864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12441822 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12441820 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47995348986368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51860024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260086196736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47495815665664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30788136 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47496066583552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21047016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20455263 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 $ _timestamp‚1.788530369069955e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30788136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47995372926464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2302022989824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20454572 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50297395916288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2301733713920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259853281792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51835150 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51860024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2253462837458 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30813008 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50297082700288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45464739187426 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259853281792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229ZíMäõ2õ- _wandbruntime‚1353  _runtime‚1353' V2XöÒ 0loss/FRt3/assignment_loss‚2.0483384657847252 /loss/FBt2/detection_loss‚2.5311271051565805 .loss/FRt2/detection_loss‚2.727988544263338 /loss/FRt1/assignment_loss‚4.842385704700764 2loss/SRqqt1/assignment_loss‚2.0686456489151923 1loss/SRqqt4/assignment_loss‚1.998579635538838 1loss/FBt2/assignment_loss‚0.40685824856983444 1loss/SRqqt4/detection_loss‚3.1252123401923613 /loss/FBt3/detection_loss‚1.0317387723404428 epoch‚0 /loss/FRt3/detection_loss‚1.8476820378319214 1loss/SRqqt3/detection_loss‚0.9894655860723958 _step‚91 .loss/FRt1/detection_loss‚2.560048473186982 1loss/FBt1/assignment_loss‚0.38918396497318497 %loss/total_loss‚55.04115021502278 1loss/FBt4/assignment_loss‚0.32073827454700665 .loss/FRt4/detection_loss‚3.287479326725006 1loss/SRqqt1/detection_loss‚2.3479157501253587 /loss/FBt1/detection_loss‚2.9150242153555155 1loss/SRqqt2/assignment_loss‚2.236379847444337 2loss/SRqqt3/assignment_loss‚1.5627207184239986 0loss/SRqqt2/detection_loss‚2.457701765257737 trainer/global_step‚1149 /loss/FRt4/assignment_loss‚6.212020168304443 .loss/FBt4/detection_loss‚2.059389011302431 % _timestamp‚1.7885303690745506e+09 1loss/FBt3/assignment_loss‚0.12274529947717383 /loss/FRt2/assignment_loss‚4.951781310533223 _runtime‚1353.228179435¾‹OÈ\÷Ö 0loss/FRt3/assignment_loss‚2.0483384657847252 /loss/FBt2/detection_loss‚2.5311271051565805 .loss/FRt2/detection_loss‚2.727988544263338 /loss/FRt1/assignment_loss‚4.842385704700764 2loss/SRqqt1/assignment_loss‚2.0686456489151923 1loss/SRqqt4/assignment_loss‚1.998579635538838 1loss/FBt2/assignment_loss‚0.40685824856983444 1loss/SRqqt4/detection_loss‚3.1252123401923613 /loss/FBt3/detection_loss‚1.0317387723404428 epoch‚0 /loss/FRt3/detection_loss‚1.8476820378319214 1loss/SRqqt3/detection_loss‚0.9894655860723958 _step‚91 .loss/FRt1/detection_loss‚2.560048473186982 1loss/FBt1/assignment_loss‚0.38918396497318497 %loss/total_loss‚55.04115021502278 1loss/FBt4/assignment_loss‚0.32073827454700665 .loss/FRt4/detection_loss‚3.287479326725006 1loss/SRqqt1/detection_loss‚2.3479157501253587 /loss/FBt1/detection_loss‚2.9150242153555155 1loss/SRqqt2/assignment_loss‚2.236379847444337 2loss/SRqqt3/assignment_loss‚1.5627207184239986 0loss/SRqqt2/detection_loss‚2.457701765257737 trainer/global_step‚1149 /loss/FRt4/assignment_loss‚6.212020168304443 .loss/FBt4/detection_loss‚2.059389011302431 % _timestamp‚1.7885303690745506e+09 1loss/FBt3/assignment_loss‚0.12274529947717383 /loss/FRt2/assignment_loss‚4.951781310533223 _runtime‚1353.228179435[œÐûaŽø‚(:ƒ ÌëÔå¯Ð— gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01«»¢cù‚(:Ø ÌëÔæëÄš disk.dm-0.in‚ 3173.4453125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2933.84375( disk./.usagePercent‚78.8530091371342& disk.nvme0n1.out‚25373.94482421875 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚43.8125$ memory_percent‚85.60380935668945' proc.memory.availableMB‚ 2358.671875 network.recv‚503272# disk.dm-0.out‚25373.94482421875 disk.drbd1260.out‚0.1875 disk.sda2.out‚0* proc.memory.percent‚17.906761169433594 proc.cpu.threads‚17 network.sent‚ 1.571541e+06% disk./.usageGB‚1174.9668197631836 disk.nvme0n1.in‚ 3173.4453125 disk.sda3.in‚0( disk.nvme0c0n1.out‚25373.94482421875 disk.dm-1.out‚43.8125 cpu‚25.119533675019667 disk.sda3.out‚43.8125" disk.nvme0c0n1.in‚ 3173.4453125k­ÿwú‚(:… ÛëÔå躗 gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌS„eû‚(:Ú ÛëÔŒãÙ› cpu‚25.119813656419545 proc.cpu.threads‚17 network.recv‚509284& disk.nvme0n1.out‚25677.78466796875 disk.sda3.in‚0 disk.dm-0.in‚ 3261.40625 disk.sda1.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2933.84375* proc.memory.percent‚17.906761169433594 disk.sda.in‚0 disk.sda3.out‚ 44.171875 disk.nvme0c0n1.in‚ 3261.40625# disk.dm-0.out‚25677.78466796875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.1875( proc.memory.availableMB‚ 2358.3984375 network.sent‚ 1.621491e+06% disk./.usageGB‚1175.0058097839355 disk.sda2.out‚0 disk.nvme0n1.in‚ 3261.40625 disk.dm-1.out‚ 44.171875) disk./.usagePercent‚78.85562579014166 disk.sda.out‚ 44.171875 disk.sda2.in‚0( disk.nvme0c0n1.out‚25677.78466796875$ memory_percent‚85.60547828674316#5%¡‘ü‚(:† êëÔËט gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.479' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8958 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïóŸÄqý‚(:æ êëԵеœ network.sent‚ 1.62647e+06( disk.nvme0c0n1.out‚25891.47412109375! disk.nvme0n1.in‚ 3287.10546875& disk.nvme0n1.out‚25891.47412109375( proc.memory.availableMB‚ 2358.1328125 proc.memory.rssMB‚ 2933.84375* proc.memory.percent‚17.906761169433594% disk./.usageGB‚1175.2172737121582 disk.sda1.in‚0 disk.drbd1260.out‚ 0.19140625 disk.sda.out‚ 44.609375 proc.cpu.threads‚17 network.recv‚512475# disk.nvme0c0n1.in‚ 3287.10546875 disk.dm-0.in‚ 3287.10546875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚85.60709953308105 cpu‚25.119233324792066) disk./.usagePercent‚78.86981731179473# disk.dm-0.out‚25891.47412109375 disk.dm-1.out‚ 44.609375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 44.609375®§Šxþ‚(:í ùëÔįŽ˜ disk.dm-0.in‚ 3287.10546875 disk.dm-1.out‚ 45.18359375 disk.sda.out‚ 45.18359375! disk.nvme0n1.in‚ 3287.10546875 disk.sda3.in‚0 network.sent‚ 1.634026e+06 network.recv‚516954 disk.sda1.out‚0& disk.nvme0n1.out‚26123.22021484375$ memory_percent‚85.60259342193604) proc.memory.availableMB‚ 2358.87109375 proc.memory.rssMB‚ 2933.84375 disk.drbd1260.in‚0% disk./.usageGB‚1175.2167587280273# disk.dm-0.out‚26123.22021484375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0( disk./.usagePercent‚78.8697827507776 disk.sda3.out‚ 45.18359375* proc.memory.percent‚17.906761169433594 cpu‚25.119186932403398 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.19140625# disk.nvme0c0n1.in‚ 3287.10546875( disk.nvme0c0n1.out‚26123.220214843756¾ké‘ÿ‚(:† ùëÔó›÷— gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2712 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ>ÃN2€- _wandbruntime‚1417  _runtime‚1417¶×´Â¼ $ lr-AdamW/pg1‚0.00080985687768999 $ lr-AdamW/pg2‚0.00080985687768999 trainer/global_step‚1199 % _timestamp‚1.7885304334142268e+09 _runtime‚1417.567971471 _step‚92׈Æ‚À % _timestamp‚1.7885304334142268e+09 _runtime‚1417.567971471 _step‚92 $ lr-AdamW/pg1‚0.00080985687768999 $ lr-AdamW/pg2‚0.00080985687768999 trainer/global_step‚1199\ªIZ2ƒ- _wandbruntime‚1417  _runtime‚1417XEQëÅ'„¿O M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54066733 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45050034890000 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2356337864192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8355384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32124242 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47160996199424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2356104962560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚1417.569035215 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21327240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47399460239620 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45050017850640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2356104962560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2349671362516 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚93 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2399848451072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47399706252516 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21942491 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52437711969792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32099396 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49517334063616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47160978196480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2356337864192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32124242 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12971118 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2349442388980 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52438023101952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311132160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚1199 % _timestamp‚1.7885304334151487e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49517083159040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21942489 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54041885 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816¬eÕ`}…ÃO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49517083159040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21942491 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21942489 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47160996199424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32099396 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49517334063616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47160978196480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2356337864192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32124242 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12971118 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2349442388980 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52438023101952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311132160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚1199 % _timestamp‚1.7885304334151487e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49517083159040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21942489 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54041885 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54066733 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45050034890000 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2356337864192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8355384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 MGDeviceª4£´L StatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32124242 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47160996199424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2356104962560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚1417.569035215 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21327240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47399460239620 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45050017850640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2356104962560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2349671362516 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚93 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2399848451072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47399706252516 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21942491 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52437711969792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2358301903360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8362992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2358068988416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2351629851710 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21960766 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21960764 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2351400877966 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32126088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54111726 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54111726 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12981920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8362262 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54086852 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32150960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12981922 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 299761664 _step‚94 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49558333221888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50079571428864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45087565247424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32150960 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2358068988416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 323701760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49558584139776 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52481705842176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45087548208064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21344914 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54086852 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 % _timestamp‚1.7885304344050713e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21960766 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47200264233472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47200264233472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21960764 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚732 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50079547488768 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47438949086030 trainer/global_step‚1199 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52481382140416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47439195099134 _runtime‚1418.559383991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47200282236416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21344182 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496å[î¸Ó&ˆÍM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21344182 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2358301903360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8362992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2358068988416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2351629851710 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21960766 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21960764 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2351400877966 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32126088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54111726 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54111726 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12981920 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8362262 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54086852 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32150960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12981922 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 299761664 _step‚94 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49558333221888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50079571428864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45087565247424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32150960 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2358068988416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 323701760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49558584139776 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52481705842176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45087548208064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21344914 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54086852 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 % _timestamp‚1.7885304344050713e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21960766 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47200264233472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47200264233472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21960764 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚732 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50079547488768 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47438949086030 trainer/global_step‚1199 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52481382140416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47439195099134 _runtime‚1418.559383991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47200282236416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0^c\Œ2‰- _wandbruntime‚1418  _runtime‚1418êvÊŸZŠÔ /loss/FRt1/assignment_loss‚4.586438038410285 2loss/SRqqt3/assignment_loss‚1.6832825327382503 /loss/FRt4/detection_loss‚3.3684679597616194 epoch‚0 1loss/SRqqt1/assignment_loss‚2.148500027755896 0loss/FBt4/assignment_loss‚0.2864803884550217 1loss/SRqqt4/detection_loss‚2.8744318099582897 _step‚95 -loss/FRt1/detection_loss‚2.35971078200218 trainer/global_step‚1199 % _timestamp‚1.7885304344070477e+09 _runtime‚1418.560470949 0loss/FRt3/assignment_loss‚1.8473853522472168 /loss/FRt4/assignment_loss‚5.476731371879578 1loss/SRqqt2/detection_loss‚2.1942277980201386 /loss/FRt2/assignment_loss‚5.238311029970646 2loss/SRqqt2/assignment_loss‚2.4780146725037517 /loss/FRt3/detection_loss‚1.5183891957060676 1loss/SRqqt1/detection_loss‚2.3093367367982864 1loss/SRqqt3/detection_loss‚1.1649181249650937 /loss/FRt2/detection_loss‚2.9202786926180124 2loss/SRqqt4/assignment_loss‚2.1749417045537163 0loss/FBt1/assignment_loss‚0.6396233442093219 2loss/FBt3/assignment_loss‚0.005465658489847556 .loss/FBt2/detection_loss‚2.672454023361206 &loss/total_loss‚53.752263779160174 0loss/FBt2/assignment_loss‚0.3730661421107167 /loss/FBt1/detection_loss‚2.9311173191437354 /loss/FBt3/detection_loss‚1.3145572498440743 /loss/FBt4/detection_loss‚1.1861338236572183ž1˜­^‹Ø 1loss/SRqqt2/detection_loss‚2.1942277980201386 /loss/FRt2/assignment_loss‚5.238311029970646 2loss/SRqqt2/assignment_loss‚2.4780146725037517 /loss/FRt3/detection_loss‚1.5183891957060676 1loss/SRqqt1/detection_loss‚2.3093367367982864 1loss/SRqqt3/detection_loss‚1.1649181249650937 /loss/FRt2/detection_loss‚2.9202786926180124 2loss/SRqqt4/assignment_loss‚2.1749417045537163 0loss/FBt1/assignment_loss‚0.6396233442093219 2loss/FBt3/assignment_loss‚0.005465658489847556 .loss/FBt2/detection_loss‚2.672454023361206 &loss/total_loss‚53.752263779160174 0loss/FBt2/assignment_loss‚0.3730661421107167 /loss/FBt1/detection_loss‚2.9311173191437354 /loss/FBt3/detection_loss‚1.3145572498440743 /loss/FBt4/detection_loss‚1.1861338236572183 /loss/FRt1/assignment_loss‚4.586438038410285 2loss/SRqqt3/assignment_loss‚1.6832825327382503 /loss/FRt4/detection_loss‚3.3684679597616194 epoch‚0 1loss/SRqqt1/assignment_loss‚2.148500027755896 0loss/FBt4/assignment_loss‚0.2864803884550217 1loss/SRqqt4/detection_loss‚2.8744318099582897 _step‚95 -loss/FRt1/detection_loss‚2.35971078200218 trainer/global_step‚1199 % _timestamp‚1.7885304344070477e+09 _runtime‚1418.560470949 0loss/FRt3/assignment_loss‚1.8473853522472168 /loss/FRt4/assignment_loss‚5.476731371879578_ã®ï¢œŒ‚(:‘ ˆžëÔ˜¶÷– gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.504' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.300800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÜ/Ps‚(:è ˆžëÔИé™ disk.drbd1260.out‚ 0.1953125 disk.sda.in‚0 proc.memory.rssMB‚ 2933.84375* proc.memory.percent‚17.906761169433594 cpu‚25.1189295805262) disk./.usagePercent‚78.87091379206387# disk.nvme0c0n1.in‚ 3287.10546875# disk.dm-0.out‚26230.86572265625 disk.sda1.in‚0 disk.drbd1260.in‚0) proc.memory.availableMB‚ 2358.56640625 network.recv‚520217% disk./.usageGB‚1175.2336120605469& disk.nvme0n1.out‚26230.86572265625( disk.nvme0c0n1.out‚26230.86572265625 disk.sda.out‚ 45.5703125$ memory_percent‚85.60445308685303 proc.cpu.threads‚17! disk.nvme0n1.in‚ 3287.10546875 disk.sda3.out‚ 45.5703125 disk.dm-0.in‚ 3287.10546875 disk.dm-1.in‚0 disk.dm-1.out‚ 45.5703125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.650098e+06 disk.sda3.in‚0ÓN*wŽ‚(:… —žëÔœšß– gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9512 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0véLu‚(:ê —žëÔåãé˜ disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 46.0078125 disk.dm-0.in‚ 3287.10546875! disk.nvme0n1.in‚ 3287.10546875& disk.nvme0n1.out‚26510.74853515625# disk.nvme0c0n1.in‚ 3287.10546875# disk.dm-0.out‚26510.74853515625 disk.dm-1.out‚ 46.0078125$ memory_percent‚86.16511821746826" proc.memory.rssMB‚ 3025.4453125 network.recv‚526042) disk./.usagePercent‚78.88768510163949 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1953125 disk.sda.in‚0( disk.nvme0c0n1.out‚26510.74853515625 disk.dm-1.in‚0) proc.memory.availableMB‚ 2266.70703125 disk.sda.out‚ 46.0078125 disk.sda2.in‚0* proc.memory.percent‚18.465852737426758 cpu‚25.15128939591239 proc.cpu.threads‚17 network.sent‚ 1.68691e+06% disk./.usageGB‚1175.4835166931152 disk.sda1.in‚0KËaœ‚(:‘ ¦žëÔêÇî” gpu.0.gpu‚42.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.097' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.019399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁcN¯l‘‚(:á ¦žëÔ“©™( proc.memory.percent‚18.4689998626709% disk.nvme0n1.out‚26815.2080078125 disk.drbd1260.out‚ 0.19921875 disk.sda.out‚ 46.5234375 disk.sda2.out‚0$ memory_percent‚86.16783618927002) proc.memory.availableMB‚ 2266.26171875 cpu‚25.19568883705055 proc.cpu.threads‚17% disk./.usageGB‚1175.4600296020508 disk.sda3.in‚0 disk.sda3.out‚ 46.5234375! disk.nvme0c0n1.in‚ 3414.921875 disk.dm-1.out‚ 46.5234375 disk.sda2.in‚0 network.sent‚ 1.691788e+06 disk.nvme0n1.in‚ 3414.921875' disk.nvme0c0n1.out‚26815.2080078125 disk.dm-0.in‚ 3414.921875" disk.dm-0.out‚26815.2080078125 disk.sda1.in‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3025.9609375 network.recv‚529334) disk./.usagePercent‚78.88610886325118 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.in‚0ƒî~Y‘’‚(:† µžëÔ˜›¢— gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7424 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GlY\h“‚(:Ý µžëԸ𘛠disk.sda3.out‚46.90625$ memory_percent‚86.27979755401611 cpu‚25.23351248489376 network.sent‚ 1.700012e+06% disk./.usageGB‚1175.4596328735352! disk.nvme0c0n1.in‚ 3414.921875' disk.nvme0c0n1.out‚27040.5322265625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.19921875 disk.sda3.in‚0 disk.sda1.in‚0% disk.nvme0n1.out‚27040.5322265625) proc.memory.availableMB‚ 2247.91796875 proc.cpu.threads‚17) disk./.usagePercent‚78.88608223846761" disk.dm-0.out‚27040.5322265625 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚46.90625 disk.nvme0n1.in‚ 3414.921875# proc.memory.rssMB‚ 3044.27734375) proc.memory.percent‚18.58079433441162 network.recv‚534333 disk.dm-0.in‚ 3414.921875 disk.dm-1.out‚46.90625 disk.sda2.in‚0 disk.sda2.out‚0hGpÚd”‚(:Ù ÄžëÔÁ€²›& proc.memory.availableMB‚ 2256.53125) proc.memory.percent‚18.52595806121826! disk.nvme0c0n1.in‚ 3414.921875& disk.nvme0c0n1.out‚27368.923828125$ disk.nvme0n1.out‚27368.923828125# proc.memory.rssMB‚ 3035.29296875 cpu‚25.270484646615188 proc.cpu.threads‚17 network.sent‚ 1.704931e+06% disk./.usageGB‚1175.4831581115723 disk.dm-0.in‚ 3414.921875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 47.265625 disk.sda2.in‚0 network.recv‚537735) disk./.usagePercent‚78.88766103693128! disk.dm-0.out‚27368.923828125 disk.dm-1.out‚ 47.265625 disk.sda1.in‚0 disk.drbd1260.out‚0.203125 disk.nvme0n1.in‚ 3414.921875 disk.sda3.out‚ 47.265625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚86.22722625732422Üœ•‚(:‘ ÄžëÔŠÓÖ¦ gpu.0.gpu‚14.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚143.448' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.689600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚)Òty–bt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÓ2ïøx—bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP$­‹âd˜bl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak"trainer/global_stepÕõ p2:JPIoZ¿h™bc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JPàú9*tšbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP5–Sk›bf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPÿ8çbqœbl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JPǹ%!tbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPYA3vžbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPÌ·5IpŸbk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JP¡›!Nt bo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPìžö;i¡bd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPHélñm¢bh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JPÖ,ø+o£bj GDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current"trainer/global_step2:JPåfe®l¤bg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPCùr¥bm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP:…*’n¦bi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JPÛóòv§bq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPXà˜u¨bp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP{ÊpÅz©bu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP&nªbi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JP«&ím«bh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JPž‡æXs¬bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPÉgYf­ba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP‘ýVWl®bg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JPrf Ìn¯bi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JPs2Í—t°bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP$s±bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP‰ðåØp²bk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JP°›ãt³bo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPE ’ w´br ODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP»„cÃ]µbX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JPAè¼ôq¶bl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPÞM)‰m·bh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JP#äÇr¸bm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JPëYÀúm¹bh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP ¡Ávºbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP@=¡o»bj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JPÐI1Zu¼bp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPx™_t½bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP0ú{y¾bt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP¡6k¿bf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JP7Uö~Àby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP•9´ãeÁb` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JPdqÂbl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JPƒd}orÃbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPgjþPpÄbk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JP³ì¸>|Åbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPï:@€cÆb^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JPöíÿÌdÇb_ ·fÚba >DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPýu²ÞeÛb` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JPHÅ€øwÜbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP„%‡oÝbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JPM8ÞtÞbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚49069982889452 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175409664 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚13651556 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚54385846598656 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55706592 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚19925 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚49027056604160 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2399212239872 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2443393512960 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚1208 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚49027056604160 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚32807708 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175409664 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚401 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2399036830208 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚51426093434368 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚49069982889452 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175409664 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚13651556 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚54385846598656 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55706592 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚19925 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚49027056604160 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2399212239872 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2443393512960 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚1208 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚49027056604160 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚32807708 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175409664 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚401 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2399036830208 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚51426093434368 Fp¿bk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JPဋ„mÀbh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JPŒ'Rñ|Ábw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP$Ï£¶oÂbj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JPÖºè%lÃbg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JPßjÄbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JPªŠL–pÅbk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPƒ—ŽìlÆbg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP²Õ«sÇbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP}3<*rÈbm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPݵ«aÉb\ 9DeviceStatsMonitor.on_validation_batch_end/max_split_size"trainer/global_step2:JP|íŠ8lÊbg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JPňȷmËbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JPü³ mÌbh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JP¶Ñq“mÍbh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JPæ±>µjÎbe BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JPKä׺qÏbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPV8í†tÐbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPj¯žúvÑbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPR7iÒbd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JPÀZmÓbh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JP‹J ìxÔbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPà[âÛsÕbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP¿mÖbh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JPQ¨øsq×bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPO'ötØbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP+œQ’pÙbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPY– ÕkÚbf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JPc‚ØFjÛbe BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPf¨‡dÜb_ DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPdÙîmqébl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPf:tkêbf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JPøàÐTnëbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JPÈ…_Íqìbl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JP¼&½ríbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP±‹Rípîbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JPŽ°uïbp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPhÁ³Îkðbf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JP3ÑQYlñbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JPÎGovòbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPÑ¡0Æeób` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP˜Ènôbi FDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current"trainer/global_step2:JP[yoõbj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPž,ô¡töbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPº^v÷bq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP8ºïiøbd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JP…£·Åvùbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP ýž|úbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP¬e Qjûbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JP£]stübo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPyiœsýbn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPlHÝcþb^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JP$uÇuÿbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP5•9Àg€bb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JPËZ&tbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP¯Å“·f‚ba >DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JP¢"Voƒbj GDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated"trainer/global_step2:JPÚÌùr„bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP‹²¶˜w…br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP°Kׯw†br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP•CÏr‡bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP>/BÀtˆbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP4·u²i‰bd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JP‚^viŠbd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JP²Åg‹bb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP’Ó^°hŒbc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JP¼–ÿnbi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JP˜ÛœÈjŽbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JPL}]2- _wandbruntime‚1499  _runtime‚1499ª†aK6)°R RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966133760 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚30751 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚32836541 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175409664 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚54416682212352 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19925 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚8702975 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193412608 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 564258816 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚51456632968704 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚13663240 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚51972814839296 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚399 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 169394688 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55710955 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175409664 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚49057140281344 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚22366215 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚13663238 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚46703320945840 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560281229 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19927 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55730882 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966133760 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188856556 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚22894339 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚49057122278400 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚32816616 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚32816616 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30508 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 339145216 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚22894339 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚54416512817664 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚51972790899200 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚49096203603452 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 _runtime‚1499.937242481 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚8702576 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 145454592 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚22894341 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2399686099968 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55710955 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193412608 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2399510690304 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19927 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚32836541 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175409664 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚54416682212352 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19925 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚8702975 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193412608 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 564258816 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚51456632968704 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚13663240 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚51972814839296 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚399 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 169394688 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55710955 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175409664 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚49057140281344 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚22366215 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚13663238 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚46703320945840 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560281229 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19927 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55730882 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966133760 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188856556 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚22894339 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚49057122278400 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚32816616 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚32816616 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30508 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 339145216 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚22894339 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚54416512817664 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚51972790899200 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚49096203603452 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 _runtime‚1499.937242481 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚8702576 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 145454592 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚22894341a²Þ³’‚(:„ ÓžëÔ­è©– gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.5726 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i;2f]“‚(:Ò ÓžëÔü¬¸˜ disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚0.203125 network.sent‚ 1.709239e+06% disk./.usageGB‚1175.7343978881836 disk.sda.in‚0 disk.nvme0n1.in‚ 3414.921875 disk.nvme0n1.out‚ 27741.71875! disk.nvme0c0n1.in‚ 3414.921875 disk.dm-0.in‚ 3414.921875 disk.dm-0.out‚ 27741.71875 disk.sda2.in‚0" disk.nvme0c0n1.out‚ 27741.71875 disk.drbd1260.in‚0) proc.memory.availableMB‚ 2267.34765625 proc.cpu.threads‚17 network.recv‚542322) disk./.usagePercent‚78.90452194914388 disk.sda3.out‚ 47.6796875" memory_percent‚86.161208152771# proc.memory.rssMB‚ 3024.11328125* proc.memory.percent‚18.457722663879395 cpu‚25.309607809904797 disk.sda.out‚ 47.6796875 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 47.6796875áÝxÞ‘”‚(:† âžëÔ äµ— gpu.0.gpu‚21.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.193' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾)Ùa•‚(:Ö âžëÔšÙéš network.sent‚ 1.780611e+06% disk./.usageGB‚1175.7333755493164 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.2109375 disk.sda2.out‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 3613.80078125 disk.sda.out‚ 48.30078125 disk.sda2.in‚0! disk.nvme0n1.in‚ 3613.80078125 disk.nvme0n1.out‚ 27936.1875 disk.dm-0.out‚ 27936.1875' proc.memory.availableMB‚ 2281.734375 cpu‚25.346341971116697 proc.cpu.threads‚17 network.recv‚546750( disk./.usagePercent‚78.9044533391247 disk.drbd1260.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 3613.80078125 disk.dm-1.in‚0 disk.dm-1.out‚ 48.30078125$ memory_percent‚86.07339859008789 disk.sda3.out‚ 48.30078125! disk.nvme0c0n1.out‚ 27936.1875! proc.memory.rssMB‚ 3009.921875* proc.memory.percent‚18.371105194091797ê ]J–bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPP<ÛlJ—bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÑ”ØJ˜bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÆVN²bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP5¹ýH³bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPMntçN´bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPGý–åNµbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPFú öN¶bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP8KBÿN·bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP“S[rJ¸bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPiÒ³J¹bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPû‡0NºbI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPv©§zN»bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP蔌CH¼bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPh_¥¶G½bB validation_average_jet_accuracy"trainer/global_step2:JPC7N¾bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP*‘8SH¿bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP-ÄJÀbE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP—Î/QNÁbI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPpñ¨HÂbC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP°BªiHÃbC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPó†¹HÄbC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPǪÓóNÅbI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP•ÎåÍNÆbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPt\ä NÇbI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPÍ<5HÈbC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPº„€iJÉbE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPkOÓ¦JÊbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP2æ©NËbI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP´>¥ÅNÌbI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP¼øfã@Íb; particle/accuracy_3_of_3"trainer/global_step2:JP¦@ïÖHÎbC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP ¨¨yJÏbE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP YÍþNÐbI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPõèï)NÑbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP"š?NÒbI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPút JNÓbI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPQ8¬NÔbI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÛ(†JÕbE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP>q±NÖbI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPñH¹ìN×bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPä.ƒJØbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÀ’ð:NÙbI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP32JÚbE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP\TdNÛbI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPЂ?HÜbC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP£séJÝbE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPÄ3 HÞbC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPlGl9HßbC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP:BKQNàbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPü¸Y¡JábE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPv;À?HâbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPý HàNãbI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPñÁJäbE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPŠÄ‚£HåbC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPVØ\!NæbI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP”QáHçbC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPQ¦˜JèbE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP®jسNébI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP‡ ?ÊHêbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP:ö¡JHëbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP.Œ*JìbE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPb B?NíbI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPë¶þJîbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPsÑ5æNïbI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPí‚ñiJðbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPô­JñbE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPœ‰›JòbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¡Þ6HóbC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPç®JYJôbE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¯¢ô¹NõbI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPVL©LNöbI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPs:½N÷bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP×ã…ËHøbC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP‚ýõJùbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPDÿ HúbC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPþw@ûb; particle/accuracy_2_of_4"trainer/global_step2:JPrä0â<üb7 particle/specificity"trainer/global_step2:JPüüúýHýbC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP'‰àJþbE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP[¤zHÿbC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPÓ>J€ bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP:,sKJ bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP2ÉOÅ@‚ b; particle/accuracy_6_of_7"trainer/global_step2:JPpµ`sHƒ bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP,2éH„ bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPv¶;ÞJ… bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP[Q#N† bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP„¼J‡ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPåþàHˆ bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP—Ž H‰ bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPiíHŠ bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPG©ÿ–H‹ bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP¤ÿqNŒ bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP$óæèN bI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP›ÂÑfNŽ bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP_>YJ bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPw’EJ bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPrõ5äJ‘ bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP²ÞÜ J’ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPÒ Á¾H“ bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP&N” bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPcDöJ• bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPdH– bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP©ÉN— bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPSàp5H˜ bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÇ\áIJ™ bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPêuÑÔJš bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPR“èjN› bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPP¹· Nœ bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP¢~æòH bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP}OÙHž bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPˆûÔJŸ bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPUÝBsJ  bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPøìùâ;¡ b6 jet/accuracy_4_of_7"trainer/global_step2:JPýˆ÷vJ¢ bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPäÚ®J£ bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP—4‘N¤ bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP—dN¥ bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP<Õ gJ¦ bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPz#N§ bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPº°‹|N¨ bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP›”jrN© bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPsÎj£Nª bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPñ›¾N« bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPe.H;H¬ bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPVgî];­ b6 jet/accuracy_1_of_1"trainer/global_step2:JP8ÎÎH® bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP:B=˜J¯ bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP"oìH° bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP ¾æzH± bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP!/u‡N² bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPkØ÷N³ bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPæf‡ ;´ b6 jet/accuracy_4_of_6"trainer/global_step2:JP/Ðø;µ b6 jet/accuracy_6_of_7"trainer/global_step2:JP9ß+±J¶ bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÀy’ N· bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÎ1HH¸ bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP æ‡'N¹ bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÚoÞHº bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP„VÍG;» b6 jet/accuracy_2_of_7"trainer/global_step2:JP­“(J¼ bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÚ²F6J½ bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP,âŽðH¾ bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPŽH¿ bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPˆ:¿:HÀ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPÍð]ÓNÁ bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP§ë"ìJ bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPñ/ùÏHà bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP5jqÃNÄ bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP rTNÅ bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPp,ƒ<Æ b7 particle/sensitivity"trainer/global_step2:JP‘áÔ¼@Ç b; particle/accuracy_4_of_7"trainer/global_step2:JP?v¯ÄJÈ bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP„NÉ bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPw€IÓ;Ê b6 jet/accuracy_1_of_3"trainer/global_step2:JPš ‡ÀJË bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPDÝñJÌ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPx HÍ bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPUÅs?NÎ bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPnçNÏ bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP^ùÉHÐ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP_ÁÂNÑ bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP¨Å…€JÒ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP"QCJÓ bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPë&ŠHÔ bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP:?NnNÕ bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP¯7¡—JÖ bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPƒGvJ× bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP}ñ ;HØ bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP±·ªNÙ bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÍÀN0HÚ bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPö&ž–JÛ bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPr*HÜ bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÒ,âJÝ bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¸xSHÞ bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPv¯ºNß bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP, &Jà bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÕmJá bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPCyhiJâ bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÄLŠ¿Jã bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÈ°sÁHä bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP-ŒNå bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPã¬%„Næ bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPå^~àHç bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP®„oÖJè bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPæ­8™Né bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÿ;¼LHê bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPƒký ;ë b6 jet/accuracy_5_of_6"trainer/global_step2:JPÕRïJì bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPzè·Ní bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPòãùÑJî bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP9¨¶ÃJï bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP·%[›Hð bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPVÚ@Ë;ñ b6 jet/accuracy_2_of_6"trainer/global_step2:JP<ô²ÍHò bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPC³KHó bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPªNô bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPŒquJõ bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPRŸ%.Nö bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPgX«N€ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPQÿŸH bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPÌmÚ¹J‚ bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP0›;ÁHƒ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPœ³¦H„ bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¯rS¤H… bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPœ=¨“H† bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPx©*#H‡ bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPT® [Jˆ bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPXý5}N‰ bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP ¤ZNŠ bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPçõvN‹ bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP]fJŒ bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPïC¿õH bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP 4fNŽ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPåÁrVN bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP.jã•N bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP[Eü J‘ bE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPŠ±açN’ bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP½^¹cJ“ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPì¬ZJ” bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÄæµ N• bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPqû;N– bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPË#/XH— bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPŸ.žJ˜ bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPZ7­J™ bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP:]–Jš bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP r¬OJ› bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÚ~µGHœ bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP·šuw; b6 jet/accuracy_2_of_3"trainer/global_step2:JP /OHž bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP#ãKJŸ bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPkêuH  bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPo}ÁøN¡ bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPžØTQJ¢ bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP´hC^H£ bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP‘FJN¤ bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP~ñ@¥ b; particle/accuracy_1_of_3"trainer/global_step2:JPùC×H¦ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP¬Û8J§ bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP»W¦l@¨ b; particle/accuracy_5_of_6"trainer/global_step2:JP$´ÃH© bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPדèJª bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPh´Ñ1N« bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP 9¯VJ¬ bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP‡’ïÅH­ bC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP‡!ªþN® bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÚþMJ¯ bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPFP{áJ° bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPË­ÇVJ± bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPìÜøuJ² bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP‹“«N³ bI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPª³±N´ bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP5¬Î¶Hµ bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPîz}AH¶ bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP`ø;ñJ· bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP“ÙÄH¸ bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP³§W+N¹ bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP=—½~Hº bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPçÍfH» bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP*ÿJ¼ bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP;5ÒJ½ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP§Ó¦«J¾ bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP-%Qå-¿ bI &Purity/2FRt*SRqqt2FBt/event_proportionÔ\ßð!"trainer/global_step2:JP_G|­HÀ bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP.œ]âHÁ bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP©¿ÊH bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPýÁõaJà bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPÐ0wNÄ bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP6_c1HÅ bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP¾tDNÆ bI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPˆÁmJÇ bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¨>À¥JÈ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP r¬u;É b6 jet/accuracy_4_of_4"trainer/global_step2:JPßÜcJÊ bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPrÕý'HË bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPPÄrJÌ bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP“·NÍ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPäª*ßHÎ bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPAlÀJÏ bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP_Jª JÐ bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP¾^;Ñ b6 jet/accuracy_5_of_7"trainer/global_step2:JP™pÚJÒ bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¨ßŒHÓ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPA 7ÛJÔ bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP)—fHÕ bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÁÐo+NÖ bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPõ ;àN× bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP 1®JØ bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPÂSèšHÙ bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPô‘ÌJÚ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPaIÜJÛ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP³¬jƒNÜ bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPBHÝ bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP‡``äJÞ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP³ÎœJß bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPzE›Jà bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPQ£v~Já bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPt%5Nâ bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPñ+ DJã bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP¥«]OJä bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPRĈkHå bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPxœÎJæ bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP‡xÍÅHç bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP„õ„Nè bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPú%ÚINé bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP›(Hê bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPb©Jë bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP£½é2Nì bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP1)ÿJí bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP±ëŸªJî bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPY©%9Jï bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÑŽ¼Hð bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP¿€—ŠNñ bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPO‰“Nò bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP½_mNó bI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP6ÙWJÿ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPérXiN€ bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP,^VJ bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPž½ŸLH‚ bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPQ(+¶Jƒ bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPVP¨¶J„ bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP+¬rJ… bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPп—[H† bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP“êíoJ‡ bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP}sð‹Jˆ bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPï^òH‰ bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPoÖùHŠ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPð)@bN‹ bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP<ÞLJŒ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPýDÚ N bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÉ rãJŽ bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPðw†§H bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP)PMÎN bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPY€¬J‘ bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP¯ ˆJ’ bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPnu ]H“ bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÔ‹¨N” bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÉ s7J• bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPQÃÇÕN– bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPóp¶N— bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP“¡m×;˜ b6 loss/val_total_loss"trainer/global_step2:JP¶ˆúkN™ bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPçFœÿNš bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPU2Å@› b; particle/accuracy_4_of_5"trainer/global_step2:JPCÓuóHœ bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPAá£H bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP& ·#Hž bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPw5ö\JŸ bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP€zÔÆJ  bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP”a´*H¡ bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP'n²;¢ b6 jet/accuracy_1_of_2"trainer/global_step2:JPn{J£ bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP5E4±H¤ bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP·h=ºN¥ bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPªÅÉÉH¦ bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP Æ›&@§ b; particle/accuracy_3_of_6"trainer/global_step2:JP2 ÕJ¨ bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP.˜©ÇJ© bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPï2dHª bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÈ›ðäN« bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP06øJ¬ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPÃXçò@­ b; particle/accuracy_1_of_6"trainer/global_step2:JP ŽJ® bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPtÔdäJ¯ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‹£¬úJ° bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP=×ßN± bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPèÝN² bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP+ ôÄH³ bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPCBó¦J´ bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP>ÆaëHµ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPO"ªN¶ bI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPÏ£Ó-N· bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP ÞŠ;¸ b6 jet/accuracy_3_of_4"trainer/global_step2:JPtßþH¹ bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPš*É@º b; particle/accuracy_7_of_7"trainer/global_step2:JPiŠaþJ» bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP½éhH¼ bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPiˆ7 H½ bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP©êò‰N¾ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPø~~9J¿ bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP´vŠJÀ bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP"~>©HÁ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP™Nì”N bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP)/óHà bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP•# NÄ bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPɶ5¾HÅ bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP‘wHÆ bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPv"‹HÇ bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP’£ŠJÈ bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP>BSfJÉ bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÝuÂJÊ bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÐÎPNË bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP åò\;Ì b6 jet/accuracy_1_of_5"trainer/global_step2:JPµš2p@Í b; particle/accuracy_3_of_7"trainer/global_step2:JP§y)HÎ bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP« óþHÏ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPï¡dNÐ bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP+%¹HÑ bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPŠúè<;Ò b6 jet/accuracy_1_of_7"trainer/global_step2:JPnd%yJÓ bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPvÆ/¦JÔ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP+#ñHÕ bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP¿çsIHÖ bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP`sçÖJ× bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPð7JØ bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP·²]VJÙ bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP›ŠØCHÚ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPàT»ÜJÛ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPe•JÜ bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPäö,ÀHÝ bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP&CIJÞ bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPxƒy0Nß bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP–ò“‡Nà bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPª·€Há bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPyÚèJâ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPMöJã bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÔëaHä bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP©ÛÍHå bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPÚñ*gJæ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPhüíNç bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP-ø[Nè bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPýžÎ8é b3 particle/f_score"trainer/global_step2:JP¼´ˆ Jê bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPì’ÿÆJë bE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPÓ_b~Hì bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP¶Þ2åJí bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP>;ZGHî bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPŠ3ˆaHï bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP¬.=úNð bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP`¯×Nñ bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP¥ZúJò bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP© ÇlHó bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPž'1¹Jô bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÕ²[YNõ bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPs ÛêJö bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP>¾’H÷ bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÛ¡0·Nø bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPT¦‡JJù bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPü“›êHú bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP±¸ÂòNû bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPƒ{é;ü b6 jet/accuracy_2_of_2"trainer/global_step2:JPÇèÔ¨Ný bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÅýòkJþ bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP² ”Jÿ bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP§Ú¨¡N€ bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPߎ€H bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP¯ÌB";‚ b6 jet/accuracy_1_of_6"trainer/global_step2:JPN±ÌS@ƒ b; particle/accuracy_2_of_6"trainer/global_step2:JP3Æé-@„ b; particle/accuracy_5_of_7"trainer/global_step2:JP/“VJ… bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPzkqŸJ† bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPý²¸J‡ bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPF÷nHˆ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP¦Y.N‰ bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP$S5JŠ bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÕZ¤N‹ bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP;ï’HŒ bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÒÞÚ«N bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP³9íÃJŽ bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPeŒí=N bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP'ôªûJ bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPOpKN‘ bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÒ–ÿH’ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP¢Aº¦J“ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPŽi&J” bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPͪ43;• b6 jet/accuracy_1_of_4"trainer/global_step2:JP«Ãª);– b6 jet/accuracy_5_of_5"trainer/global_step2:JPˆi3ÚJ— bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP€ÆÜH˜ bC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPxmGH™ bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPŠ)Ž*Nš bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPéÚŸ-N› bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP/Ûµ¢Jœ bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP®Ç×H bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPNÿˆÆNž bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP‘D¯NŸ bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP£`CJ  bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÍ£ ˜J¡ bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPýíÖƒH¢ bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPßc6ÄH£ bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPy­‡ÏH¤ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPl®ŠnN¥ bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPH®„J¦ bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPŽ;Þ±J§ bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPóùÞ¹N¨ bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÜU¦ÎN© bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPX1ÅNª bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP¡r N« bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPöB±J¬ bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP¢µáfN­ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP" ê5J® bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP@nkH¯ bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPxÆ|N° bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPd?H± bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP<Æ×sN² bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP‚e•N³ bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP=œ­¬H´ bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP8ã-Hµ bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPl öJ¶ bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP‚ly•N· bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP˼u±N¸ bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP2@¹ b; particle/accuracy_2_of_3"trainer/global_step2:JP*3òHº bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP;ê}¹J» bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPò¼J¼ bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP´i¡óJ½ bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPn4©ãN¾ bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP8 !N¿ bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP’qt9NÀ bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP@Uä>HÁ bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPâ/¥SH bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP„üf’Jà bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPÊJÄ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¨ ·LNÅ bI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPhPóšHÆ bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPüÅj€NÇ bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPчØøHÈ bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPyñ—~;É b6 jet/accuracy_2_of_5"trainer/global_step2:JP^‡þå@Ê b; particle/accuracy_3_of_5"trainer/global_step2:JPCD*HË bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP>2ç;Ì b6 jet/accuracy_3_of_6"trainer/global_step2:JP¢k HÍ bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPú…™NÎ bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP²Ì”JÏ bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPߣ(NÐ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP.À¼öHÑ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPRÛ@Ò b; particle/accuracy_4_of_4"trainer/global_step2:JPF’JÓ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPUÚ¡ÂJÔ bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPŽç`sNÕ bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP€2ЩNÖ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP¬ùò®N× bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÁEHØ bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPˆé™IHÙ bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPÕHà JÚ bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP˜Ð³NÛ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPNˆ›†NÜ bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÙ‰µXHÝ bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP¸4¾@Þ b; particle/accuracy_1_of_2"trainer/global_step2:JPüÇj2Hß bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP„}?Jà bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPÉhHá bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP<¨sNâ bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP?v¥âNã bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÌŽm@ä b; particle/accuracy_3_of_4"trainer/global_step2:JP©"9Jå bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPu€ÕZHæ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÛ/eJç bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPY–ÓÊNè bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP²6ôJé bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPH¿C£Hê bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÞ7Ž\Jë bE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP»/ÞJì bE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPå­òNí bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPª#<+Jî bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP±w«Hï bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPÄú¼`Hð bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPo÷:`Jñ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPÓŒHò bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP~€NdJó bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPjÚçNô bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP«X&w@õ b; particle/accuracy_1_of_1"trainer/global_step2:JPÇ—¥ãJö bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP¼©OYJ÷ bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP´¤Jø bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¿lŠvHù bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP­" Jú bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPí€GýNû bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP‚(Ñ;ü b6 jet/accuracy_6_of_6"trainer/global_step2:JP’èØHý bC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPÇ<þàNþ bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPRXtHÿ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP)n#ÿJ€ bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP2õGJ bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÊý;N‚ bI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP€<«Jƒ bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÕ=oÏN„ bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP6_ŒN… bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPËØÛRN† bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP§4ëFN‡ bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP%ž hJˆ bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP"•ÜN‰ bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP#eÏ@Š b; particle/accuracy_2_of_2"trainer/global_step2:JPÐ@Æú@‹ b; particle/accuracy_1_of_7"trainer/global_step2:JP~N4A@Œ b; particle/accuracy_2_of_7"trainer/global_step2:JPT@W-J bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP î±k;Ž b6 jet/accuracy_2_of_4"trainer/global_step2:JPpËcH bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP%!H bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPÇhçÄN‘ bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPØöÚN’ bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP£lbN“ bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPUÃcwH” bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP´$öN• bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPf‰;– b6 jet/accuracy_4_of_5"trainer/global_step2:JP©­É@— b; particle/accuracy_1_of_5"trainer/global_step2:JPÒ5êÙJ˜ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPFes…J™ bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÃ@2Jš bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPü3wH› bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÉ.…Hœ bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP”Æ6JH bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPŽ»àJž bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÿb•NŸ bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP%„ÛÚJ  bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP?‹|N¡ bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP¶´!Î@¢ b; particle/accuracy_2_of_5"trainer/global_step2:JPÞD¡H£ bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPín¦H¤ bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP8 ³J¥ bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÃÎçËJ¦ bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP÷ý ‰H§ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPìóŠwJ¨ bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP0ÞN© bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP»“Õ9ª b4 particle/accuracy"trainer/global_step2:JP ø¹;« b6 jet/accuracy_3_of_7"trainer/global_step2:JPPB úH¬ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPmÝN­ bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP+ö:(H® bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPãè ;¯ b6 jet/accuracy_7_of_7"trainer/global_step2:JPŽ J° bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPù€ðTH± bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPU67^N² bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPíÞf[H³ bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP÷éÚŸJ´ bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPðÅêJµ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¢Ù÷RH¶ bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPaJ‡J· bE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPü ƒH¸ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP"ôßN¹ bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPûM}Jº bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPôk<èN» bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP¤õA™@¼ b; particle/accuracy_6_of_6"trainer/global_step2:JP1ÏsñH½ bC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP¥fÑ>J¾ bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPWNd’H¿ bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP§×ÖNÀ bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP<Ž(¤NÁ bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP[öa¡N bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP¥Ö«Nà bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPøu;JÄ bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPyNOJÅ bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPXÐÁ›@Æ b; particle/accuracy_1_of_4"trainer/global_step2:JPoЩ·HÇ bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPòoËJÈ bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŽ6FHÉ bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPYÌyNÊ bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP†*áJË bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPý…·HÌ bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPË™; NÍ bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP ˆ‡JÎ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP¦ÊÒJÏ bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP,çÓTNÐ bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP€½ŠNÑ bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP—“þ$JÒ bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP4+KHÓ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPq øÔJÔ bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPh\.åNÕ bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP ¹~ÄJÖ bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP}L{À4× bE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/gëþâqlobal_step2:JPiÇ$ÞJØ bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPæNÙ bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPÒ1-ÕNÚ bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPQI_ÓJÛ bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPËw¿.JÜ bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP»‚t;Ý b6 jet/accuracy_3_of_3"trainer/global_step2:JP²Ÿ%.HÞ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPSáo)Jß bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPèå·ZNà bI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPŸ¸:Ná bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP§^F*Nâ bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPø@Hã bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP’-ÙJä bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPü9Ì2å - _wandbruntime‚1517  _runtime‚1517Á#-w…{æ ¯¤ 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 )jet/accuracy_1_of_2‚0.679610013961792 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3400000035762787 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14848098158836365 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2765151262283325 particle/accuracy_3_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14033730328083038 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.28540298342704773 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.365521639585495 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,jet/accuracy_3_of_4‚0.040166616439819336 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_7_of_7‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4376152455806732 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.258479505777359 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34930554032325745 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2729087471961975 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9899343848228455 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.37171053886413574 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6187103986740112 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3720238208770752 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.876555323600769 particle/accuracy_3_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1723327785730362 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3199999928474426 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09354057163000107 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04343428835272789 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08641976118087769 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004938271827995777 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 &particle/f_score‚0.340833455324173 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 +"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3684210479259491 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.30515676736831665 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.479885071516037 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3871590793132782 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9750826954841614 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_2_of_2‚0.08584421128034592 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2612369954586029 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.41999703645706177 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09333334863185883 *jet/accuracy_1_of_6‚0.9807692170143127 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_7‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11099876463413239 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3988553583621979 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2538720667362213 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.18518519401550293 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12438327819108963 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3457997143268585 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.8454501628875732 jet/accuracy_5_of_5‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9686444401741028 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3935185372829437 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 trainer/global_step‚1208 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4131944477558136 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05911390110850334 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9957671761512756 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16039305925369263 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4011715054512024 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.25648149847984314 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4065377116203308 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9813914895057678 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10207546502351761 )jet/accuracy_2_of_5‚0.323233425617218 particle/accuracy_3_of_5‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.03846153989434242 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.26797208189964294 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13209731876850128 particle/accuracy_4_of_4‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.08237548172473907 particle/accuracy_1_of_2‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2941175401210785 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.30187076330184937 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.44320040941238403 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9854267239570618 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8170139193534851 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05477789416909218 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08958522230386734 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011130007915198803 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17129558324813843 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.32829147577285767 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.19752758741378784 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_7‚1 particle/accuracy_2_of_7‚1 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.243390753865242 *jet/accuracy_2_of_4‚0.2893490195274353 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.006496415939182043 particle/accuracy_1_of_5‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3975749909877777 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.23456792533397675 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957014322280884 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.42414262890815735 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.008364757522940636 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9968306422233582 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11023422330617905 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9886167049407959 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.42692074179649353 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8360459804534912 jet/accuracy_3_of_7‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10693342238664627 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 jet/accuracy_7_of_7‚0 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.425566166639328 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1349475383758545 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.44985276460647583 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11989796161651611 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.23333333432674408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9830693602561951 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 particle/accuracy_6_of_6‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.42955973744392395 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.09770888835191727 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9259260296821594 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3581443428993225 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.184523805975914 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.42097008228302 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10890120267868042 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3409820795059204 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0408707819879055 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17302417755126953 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3761206269264221 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5075757503509521 +jet/accuracy_3_of_3‚0.01932600885629654 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3304263651371002 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2126644402742386 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 epoch‚0 particle/accuracy_5_of_5‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14814816415309906 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9874974489212036 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13489976525306702 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1616959124803543 6validation_average_jet_accuracy‚0.3835704028606415 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973955154418945 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3808116614818573 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 particle/accuracy_3_of_3‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.48148149251937866 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.017149295657873154 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3553161025047302 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14622505009174347 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _step‚98 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.28125 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9880911707878113 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9676723480224609 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11929813027381897 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4627267122268677 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.006451612804085016 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020877351984381676 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2753232717514038 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.258708655834198 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ( Purity/2FRt4SRqqt0FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.34913769364356995 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.08207070827484131 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9404551982879639 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3784736096858978 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.003951365128159523 particle/accuracy_6_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13738004863262177 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9875690340995789 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9864476323127747 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4390861690044403 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9856187701225281 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.18518517911434174 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6615528464317322 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3974907696247101 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.37719300389289856 jet/accuracy_4_of_7‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07587900012731552 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.40357592701911926 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 +jet/accuracy_1_of_1‚0.42494282126426697 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3124999701976776 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993850588798523 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11509894579648972 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_6‚0 jet/accuracy_6_of_7‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10102526098489761 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10602869093418121 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14075729250907898 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3547758162021637 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09209998697042465 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +particle/sensitivity‚0.2717694640159607 particle/accuracy_4_of_7‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 *jet/accuracy_1_of_3‚0.7980836629867554 +"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9806202054023743 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3584514260292053 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21810343861579895 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3750150799751282 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9794280529022217 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ( Purity/2FRt4SRqqt*FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9783239960670471 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3249668478965759 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4383526146411896 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3503830134868622 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.07319819927215576 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9600694179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3860543966293335 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009594242088496685 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 +jet/accuracy_2_of_6‚0.32692307233810425 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.41304194927215576 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10030575841665268 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10102246701717377 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.388534814119339 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3880479633808136 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9225146770477295 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4662698209285736 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 % _timestamp‚1.7885305338372796e+09 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 _runtime‚1517.993214262 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.44613590836524963 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.18404993414878845 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4057156443595886 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4070758819580078 particle/accuracy_5_of_6‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4070758819580078 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1706896722316742 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07006028294563293 :"P­;ô±urity/1FRt*SRqqt2FBt/event_purity‚0.07188760489225388 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9946901202201843 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.41904670000076294 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3046295940876007 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.23838837444782257 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13990917801856995 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2130051702260971 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2757440209388733 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.39103370904922485 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_4‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2675034999847412 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.26323720812797546 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024420025292783976 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 jet/accuracy_5_of_7‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.43914201855659485 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1522904634475708 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.30342260003089905 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.23021437227725983 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1632039099931717 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03219766914844513 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15539146959781647 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13218598067760468 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03095238097012043 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4217810332775116 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.42273613810539246 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_5‚0.036738354712724686 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013542456552386284 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18823720514774323 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25729385018348694 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9833806753158569 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9838884472846985 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.349784791469574 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4245356321334839 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4345843493938446 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3518848717212677 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.32507696747779846 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08958522230386734 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20960676670074463 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992612600326538 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 *loss/val_total_loss‚52.435966259750316 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9288195371627808 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.11363636702299118 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9887008666992188 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5378788113594055 ("Purity/3FRt1SRqqt0FBt/event_purity‚0pv4Aiç ³¤ 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07006028294563293 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07188760489225388 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9946901202201843 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.41904670000076294 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3046295940876007 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.23838837444782257 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13990917801856995 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2130051702260971 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2757440209388733 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.39103370904922485 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_4‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2675034999847412 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.26323720812797546 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024420025292783976 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 jet/accuracy_5_of_7‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.43914201855659485 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1522904634475708 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.30342260003089905 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.23021437227725983 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1632039099931717 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03219766914844513 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15539146959781647 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13218598067760468 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03095238097012043 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4217810332775116 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.42273613810539246 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_5‚0.036738354712724686 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013542456552386284 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18823720514774323 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25729385018348694 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9833806753158569 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9838884472846985 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.349784791469574 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4245356321334839 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4345843493938446 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3518848717212677 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.32507696747779846 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08958522230386734 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20960676670074463 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992612600326538 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 *loss/val_total_loss‚52.435966259750316 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9288195371627808 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.11363636702299118 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9887008666992188 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5378788113594055 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 )jet/accuracy_1_of_2‚0.679610013961792 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3400000035762787 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14848098158836365 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2765151262283325 particle/accuracy_3_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14033730328083038 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.28540298342704773 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.365521639585495 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,jet/accuracy_3_of_4‚0.040166616439819336 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_7_of_7‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4376152455806732 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.258479505777359 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34930554032325745 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2729087471961975 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9899343848228455 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.37171053886413574 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6187103986740112 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3720238208770752 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.876555323600769 particle/accuracy_3_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1723327785730362 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3199999928474426 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09354057163000107 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04343428835272789 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08641976118087769 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004938271827995777 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 &particle/f_score‚0.340833455324173 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 +"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3684210479259491 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.30515676736831665 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.479885071516037 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3871590793132782 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9750826954841614 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_2_of_2‚0.08584421128034592 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2612369954586029 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.41999703645706177 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09333334863185883 *jet/accuracy_1_of_6‚0.9807692170143127 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_7‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11099876463413239 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3988553583621979 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2538720667362213 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.18518519401550293 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12438327819108963 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3457997143268585 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.8454501628875732 jet/accuracy_5_of_5‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9686444401741028 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3935185372829437 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 trainer/global_step‚1208 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4131944477558136 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05911390110850334 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9957671761512756 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16039305925369263 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4011715054512024 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.25648149847984314 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4065377116203308 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9813914895057678 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10207546502351761 )jet/accuracy_2_of_5‚0.323233425617218 particle/accuracy_3_of_5‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.03846153989434242 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.26797208189964294 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13209731876850128 particle/accuracy_4_of_4‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.08237548172473907 particle/accuracy_1_of_2‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2941175401210785 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.30187076330184937 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.44320040941238403 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9854267239570618 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8170139193534851 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05477789416909218 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08958522230386734 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011130007915198803 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17129558324813843 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.32829147577285767 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.19752758741378784 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_7‚1 particle/accuracy_2_of_7‚1 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.243390753865242 *jet/accuracy_2_of_4‚0.2893490195274353 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.006496415939182043 particle/accuracy_1_of_5‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3975749909877777 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.23456792533397675 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957014322280884 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.42414262890815735 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.008364757522940636 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9968306422233582 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11023422330617905 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9886167049407959 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.42692074179649353 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8360459804534912 jet/accuracy_3_of_7‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10693342238664627 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 jet/accuracy_7_of_7‚0 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.425566166639328 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1349475383758545 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.44985276460647583 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11989796161651611 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.23333333432674408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9830693602561951 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 particle/accuracy_6_of_6‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.42955973744392395 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.09770888835191727 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9259260296821594 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3581443428993225 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.184523805975914 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.42097008228302 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10890120267868042 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3409820795059204 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0408707819879055 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17302417755126953 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3761206269264221 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5075757503509521 +jet/accuracy_3_of_3‚0.01932600885629654 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3304263651371002 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2126644402742386 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 epoch‚0 particle/accuracy_5_of_5‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14814816415309906 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9874974489212036 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13489976525306702 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1616959124803543 6validation_average_jet_accuracy‚0.3835704028606415 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973955154418945 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3808116614818573 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 particle/accuracy_3_of_3‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.48148149251937866 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.017149295657873154 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3553161025047302 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14622505009174347 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _step‚98 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ."Purity/1FRt3SRqqt1FBt/˜ai ù(SRqqt_purity‚0.28125 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9880911707878113 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9676723480224609 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11929813027381897 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4627267122268677 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.006451612804085016 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020877351984381676 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2753232717514038 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.258708655834198 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ( Purity/2FRt4SRqqt0FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.34913769364356995 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.08207070827484131 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9404551982879639 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3784736096858978 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.003951365128159523 particle/accuracy_6_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13738004863262177 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9875690340995789 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9864476323127747 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4390861690044403 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9856187701225281 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.18518517911434174 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6615528464317322 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3974907696247101 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.37719300389289856 jet/accuracy_4_of_7‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07587900012731552 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.40357592701911926 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 +jet/accuracy_1_of_1‚0.42494282126426697 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3124999701976776 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993850588798523 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11509894579648972 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_6‚0 jet/accuracy_6_of_7‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10102526098489761 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10602869093418121 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14075729250907898 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3547758162021637 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09209998697042465 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +particle/sensitivity‚0.2717694640159607 particle/accuracy_4_of_7‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 *jet/accuracy_1_of_3‚0.7980836629867554 +"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9806202054023743 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3584514260292053 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21810343861579895 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3750150799751282 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9794280529022217 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ( Purity/2FRt4SRqqt*FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9783239960670471 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3249668478965759 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4383526146411896 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3503830134868622 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.07319819927215576 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9600694179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3860543966293335 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009594242088496685 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 +jet/accuracy_2_of_6‚0.32692307233810425 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.41304194927215576 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10030575841665268 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10102246701717377 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.388534814119339 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3880479633808136 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9225146770477295 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4662698209285736 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 % _timestamp‚1.7885305338372796e+09 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 _runtime‚1517.993214262 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.44613590836524963 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.18404993414878845 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4057156443595886 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4070758819580078 particle/accuracy_5_of_6‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4070758819580078 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1706896722316742 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0bq'ðèœè ‚(:‘ ñžëÔÿ“ • gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚138.585' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.717000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*þJˆeé ‚(:Ú ñžëÔ€öЗ disk.drbd1260.in‚0# proc.memory.availableMB‚1929.75# proc.memory.rssMB‚ 3020.05078125# disk.nvme0n1.out‚28366.80078125 disk.sda3.in‚0% disk.nvme0c0n1.out‚28366.80078125 disk.sda.out‚ 49.078125 disk.sda2.out‚0* proc.memory.percent‚18.432927131652832 cpu‚25.352071894264384 proc.cpu.threads‚17 disk.dm-1.in‚0 network.sent‚ 1.78519e+06! disk.nvme0n1.in‚ 3613.80078125 disk.dm-0.in‚ 3613.80078125 disk.dm-0.out‚28366.80078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 network.recv‚551722 disk.dm-1.out‚ 49.078125 disk.drbd1260.out‚ 0.2109375 disk.sda2.in‚0$ memory_percent‚88.22174072265625) disk./.usagePercent‚78.94370262222168% disk./.usageGB‚1176.3182182312012 disk.sda3.out‚ 49.078125# disk.nvme0c0n1.in‚ 3613.80078125{µý/‘ê ‚(:† €ŸëÔâ½ó— gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a”Á#që ‚(:æ €ŸëÔ¯ª$ disk./.usageGB‚1176.316577911377# disk.nvme0c0n1.in‚ 3613.80078125 disk.dm-0.in‚ 3613.80078125 disk.dm-1.out‚ 49.4765625 disk.sda2.out‚0$ memory_percent‚88.22174072265625# proc.memory.rssMB‚ 3020.05078125) disk./.usagePercent‚78.94359253898196# disk.dm-0.out‚28899.32275390625 disk.dm-1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 3613.80078125& disk.nvme0n1.out‚28899.32275390625# proc.memory.availableMB‚1929.75* proc.memory.percent‚18.432927131652832 network.recv‚557402 disk.sda3.in‚0( disk.nvme0c0n1.out‚28899.32275390625 disk.sda1.in‚0 disk.drbd1260.in‚0 cpu‚25.34965222489935 proc.cpu.threads‚17 network.sent‚ 1.907711e+06 disk.sda3.out‚ 49.4765625 disk.sda1.out‚0 disk.drbd1260.out‚ 0.21484375 disk.sda.in‚0 disk.sda.out‚ 49.4765625Mÿ Õœì ‚(:‘ ŸëÔ’ûø– gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.565' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.712999999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èAƒ|í ‚(:ñ ŸëÔ‰Ãíš& disk.nvme0n1.out‚29203.15869140625# disk.dm-0.out‚29203.15869140625$ memory_percent‚88.22333812713623# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 proc.cpu.threads‚17 network.recv‚561926 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 3613.80078125) proc.memory.availableMB‚ 1929.48828125 network.sent‚ 1.913711e+06 disk.dm-1.out‚ 49.85546875 disk.sda.out‚ 49.85546875 disk.sda3.in‚0 disk.sda3.out‚ 49.85546875 cpu‚25.347384736164738 disk.sda2.out‚0) disk./.usagePercent‚78.94359381901963% disk./.usageGB‚1176.3165969848633 disk.dm-1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 3613.80078125( disk.nvme0c0n1.out‚29203.15869140625 disk.dm-0.in‚ 3613.80078125 disk.drbd1260.out‚ 0.21484375 disk.sda.in‚0æ²õ&2î - _wandbruntime‚1570  _runtime‚1570Åíì~Æï À & lr-AdamW/pg1‚0.0008165840846387016 & lr-AdamW/pg2‚0.0008165840846387016 trainer/global_step‚1249 % _timestamp‚1.7885305861970172e+09 _runtime‚1570.350626696 _step‚99wèäÊð Ä & lr-AdamW/pg2‚0.0008165840846387016 trainer/global_step‚1249 % _timestamp‚1.7885305861970172e+09 _runtime‚1570.350626696 _step‚99 & lr-AdamW/pg1‚0.0008165840846387016c~ÂÇ2ñ - _wandbruntime‚1570  _runtime‚1570(žízÆ'ò ÀO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34026106 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54034876438016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54034852497920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51013940274789 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚1249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23825305 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2530064465920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14247186 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48536810688016 _step‚100 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53503607997952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51019417523200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2484190474752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23825307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53503859668480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23825307 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33999764 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9046537 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2484424142336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57851413 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57825069 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2484424142336 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2477375607209 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53503607997952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51019435526144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2484190474752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56564940903936 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51014186295225 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚1570.353512024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14247184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23825305 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23294410 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56564615857664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51019435526144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51019417523200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34026106 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885305861984184e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33999764 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53503859668480 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53503607997952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51019417523200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2484190474752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23825307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53503859668480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23825307 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33999764 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9046537 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2484424142336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57851413 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57825069 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2484424142336 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2477375607209 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53503607997952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51019435526144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2484190474752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56564940903936 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51014186295225 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚1570.353512024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14247184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23825305 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23294410 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56564615857664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51019435526144 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51019417523200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 N?DeviceStatsMonitor.on_traÎó in_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34026106 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885305861984184e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33999764 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53503859668480 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14257986 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51053429121199 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48574324006080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34026456 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2486388181504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2486154500608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23311694 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51058721563136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9053708 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51058703560192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54076560408064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7885305875330412e+09 _runtime‚1571.687741586 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34026456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56608295181824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23843580 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9054362 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56608609729024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57870036 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57870036 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57896406 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2486388181504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531758713856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54076536467968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48574341045440 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23843582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23843582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23312350 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51053675141843 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53545109744640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51058721563136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57896406 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53544858060800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51058703560192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2479105115119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34052824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2532049320960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2479334096403 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34052824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¸jâÔ&ö ÎM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51058703560192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54076560408064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7885305875330412e+09 _runtime‚1571.687741586 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34026456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56608295181824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23843580 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9054362 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56608609729024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57870036 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57870036 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57896406 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2486388181504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531758713856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54076536467968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48574341045440 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23843582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23843582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23312350 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51053675141843 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53545109744640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51058721563136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57896406 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53544858060800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51058703560192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2479105115119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34052824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2532049320960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2479334096403 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34052824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14257986 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51053429121199 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48574324006080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34026456 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2486388181504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2486154500608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23311694 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51058721563136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9053708 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273eF'•‚2÷ - _wandbruntime‚1571  _runtime‚1571´ánÚYø Ó 0loss/SRqqt2/detection_loss‚2.509213561011899 /loss/FBt1/detection_loss‚2.4020797820652233 epoch‚1 0loss/FBt1/assignment_loss‚0.3957690920892473 3loss/FBt3/assignment_loss‚0.0007242244802370124 1loss/SRqqt3/detection_loss‚0.9489297114667439 _runtime‚1571.689694928 _step‚102 0loss/FRt3/assignment_loss‚1.8771751406716137 /loss/FRt4/assignment_loss‚4.983057336373762 2loss/SRqqt2/assignment_loss‚2.0132497875921187 .loss/FRt4/detection_loss‚3.688486700708216 %loss/total_loss‚51.26717893580877 2loss/SRqqt1/assignment_loss‚1.6257231028341665 2loss/SRqqt3/assignment_loss‚1.3417972614899987 .loss/FRt1/detection_loss‚2.504711616784334 .loss/FRt3/detection_loss‚1.754069011385848 /loss/FRt1/assignment_loss‚4.482342921197414 1loss/SRqqt4/assignment_loss‚1.781864786254508 0loss/FBt2/assignment_loss‚0.5766308973228055 % _timestamp‚1.7885305875359979e+09 1loss/FBt4/assignment_loss‚0.28619362207983656 .loss/FRt2/detection_loss‚2.639972223554339 0loss/SRqqt4/detection_loss‚3.045538508466312 /loss/FBt2/detection_loss‚2.7737511272231736 /loss/FBt4/detection_loss‚1.7601756176857655 /loss/FRt2/assignment_loss‚4.722205359595162 trainer/global_step‚1249 1loss/SRqqt1/detection_loss‚2.3365162829073465 /loss/FBt3/detection_loss‚0.8170012605686983zï2Ç]ù × % _timestamp‚1.7885305875359979e+09 1loss/FBt4/assignment_loss‚0.28619362207983656 .loss/FRt2/detection_loss‚2.639972223554339 0loss/SRqqt4/detection_loss‚3.045538508466312 /loss/FBt2/detection_loss‚2.7737511272231736 /loss/FBt4/detection_loss‚1.7601756176857655 /loss/FRt2/assignment_loss‚4.722205359595162 trainer/global_step‚1249 1loss/SRqqt1/detection_loss‚2.3365162829073465 /loss/FBt3/detection_loss‚0.8170012605686983 0loss/SRqqt2/detection_loss‚2.509213561011899 /loss/FBt1/detection_loss‚2.4020797820652233 epoch‚1 0loss/FBt1/assignment_loss‚0.3957690920892473 3loss/FBt3/assignment_loss‚0.0007242244802370124 1loss/SRqqt3/detection_loss‚0.9489297114667439 _runtime‚1571.689694928 _step‚102 0loss/FRt3/assignment_loss‚1.8771751406716137 /loss/FRt4/assignment_loss‚4.983057336373762 2loss/SRqqt2/assignment_loss‚2.0132497875921187 .loss/FRt4/detection_loss‚3.688486700708216 %loss/total_loss‚51.26717893580877 2loss/SRqqt1/assignment_loss‚1.6257231028341665 2loss/SRqqt3/assignment_loss‚1.3417972614899987 .loss/FRt1/detection_loss‚2.504711616784334 .loss/FRt3/detection_loss‚1.754069011385848 /loss/FRt1/assignment_loss‚4.482342921197414 1loss/SRqqt4/assignment_loss‚1.781864786254508 0loss/FBt2/assignment_loss‚0.5766308973228055f¶›­œú ‚(:‘ žŸëÔ²ºý• gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.952' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.790399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÎLiû ‚(:Þ žŸëÔ¢‹›œ disk.dm-1.in‚0* proc.memory.percent‚18.432974815368652 cpu‚25.345191525329103 network.sent‚ 1.918577e+06 disk.dm-0.in‚ 3777.09765625 disk.dm-0.out‚ 29700.6484375! disk.nvme0n1.in‚ 3777.09765625 disk.sda3.out‚ 50.3046875# disk.nvme0c0n1.in‚ 3777.09765625$ memory_percent‚88.22896480560303 proc.cpu.threads‚17 network.recv‚565411( disk./.usagePercent‚78.9435927949895 disk.dm-1.out‚ 50.3046875 disk.sda1.out‚0 disk.sda.in‚0" disk.nvme0n1.out‚ 29700.6484375$ disk.nvme0c0n1.out‚ 29700.6484375) proc.memory.availableMB‚ 1928.56640625 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.21875 disk.sda.out‚ 50.3046875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 3020.05859375% disk./.usageGB‚1176.3165817260742™±ô£cü ‚(:Ø ­ŸëÔ´¶Ð˜% disk.nvme0c0n1.out‚29937.92578125 disk.dm-0.out‚29937.92578125 disk.dm-1.in‚0 disk.dm-1.out‚ 50.8046875 disk.sda1.out‚0( disk./.usagePercent‚78.9435927949895% disk./.usageGB‚1176.3165817260742 disk.drbd1260.in‚0$ memory_percent‚88.22917938232422& proc.memory.availableMB‚ 1928.53125# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 cpu‚25.3428925980613 network.recv‚572313 disk.sda.in‚0 disk.drbd1260.out‚0.21875 proc.cpu.threads‚17 network.sent‚ 2.02915e+06 disk.sda.out‚ 50.8046875 disk.sda2.out‚0 disk.nvme0n1.in‚ 3801.3671875# disk.nvme0n1.out‚29937.92578125" disk.nvme0c0n1.in‚ 3801.3671875 disk.dm-0.in‚ 3801.3671875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 50.8046875s–À]‘ý ‚(:† ­ŸëÔ¦Ú‹˜ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.059' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2118 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ݾ@í›þ ‚(: ¼ŸëÔ€°Ã• gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚138.22' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.644000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^hEuÿ ‚(:ê ¼ŸëÔ”®Â˜ disk.dm-1.in‚0 disk.nvme0n1.in‚ 3801.3671875 disk.sda3.in‚0 disk.sda3.out‚ 51.21484375 disk.dm-1.out‚ 51.21484375 disk.sda.out‚ 51.21484375( proc.memory.availableMB‚ 1927.9765625# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 proc.cpu.threads‚17 disk.sda1.in‚0) disk./.usagePercent‚78.97729029869895% disk./.usageGB‚1176.8186988830566 disk.dm-0.in‚ 3801.3671875# disk.dm-0.out‚30073.86669921875& disk.nvme0n1.out‚30073.86669921875 network.recv‚575709 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.22265625 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚88.23256492614746 cpu‚25.340675655822 disk.sda1.out‚0 network.sent‚ 2.032215e+06" disk.nvme0c0n1.in‚ 3801.3671875( disk.nvme0c0n1.out‚30073.86669921875u®x‘€‚(:† ËŸëÔû¡¬– gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.971' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3942 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v| ds‚(:è ËŸëÔ—üΘ# disk.dm-0.out‚30429.96826171875$ memory_percent‚88.23890686035156% proc.memory.availableMB‚ 1926.9375% disk./.usageGB‚1176.8166999816895 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 3801.3671875 disk.dm-0.in‚ 3801.3671875 disk.drbd1260.out‚ 0.22265625# proc.memory.rssMB‚ 3020.05859375 network.sent‚ 2.038311e+06' disk./.usagePercent‚78.977156150751& disk.nvme0n1.out‚30429.96826171875 disk.sda3.out‚ 51.55078125 disk.dm-1.out‚ 51.55078125 disk.sda1.in‚0* proc.memory.percent‚18.432974815368652 cpu‚25.338643990242396 network.recv‚580090 disk.sda3.in‚0( disk.nvme0c0n1.out‚30429.96826171875 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 51.55078125 disk.nvme0n1.in‚ 3801.3671875f§;,2‚- _wandbruntime‚1628  _runtime‚1628©—´ÇƒÁ & lr-AdamW/pg1‚0.0008164734727983567 & lr-AdamW/pg2‚0.0008164734727983567 trainer/global_step‚1299 % _timestamp‚1.7885306445020108e+09 _runtime‚1628.655465188 _step‚103z•ñŒË„Å % _timestamp‚1.7885306445020108e+09 _runtime‚1628.655465188 _step‚103 & lr-AdamW/pg1‚0.0008164734727983567 & lr-AdamW/pg2‚0.0008164734727983567 trainer/global_step‚1299gfQ62…- _wandbruntime‚1628  _runtime‚1628¼çP{À†ÀO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9393567 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 trainer/global_step‚1299 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24180851 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55566125554176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24739057 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55566377224704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14787286 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58748601539584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _b!!step‚104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35337716 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚52983737375744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55566125554176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2582406181376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52983719372800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35364058 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885306445029755e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50413328559216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚1628.656621398 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14787284 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2575084666641 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35364058 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60103115 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629550539264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24181552 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24739055 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52983719372800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35364058 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885306445029755e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50413328559216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚1628.656621398 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14787284 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2575084666641 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35364058 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60103115 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629550539264 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24181552 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24739055 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24180851 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55566125554176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24739057 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55566377224704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14787286 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58748601539584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35337716 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚52983737375744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55566125554176hÏ™}Z2ˆ- _wandbruntime‚1629  _runtime‚1629&)̈Ð&‰ÊM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53023023412736 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58792282591232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2584370207232 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55607375617024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53028131053551 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50450841877280 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2584603888128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24198682 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56160758910464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35390776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24757332 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53023023412736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2584370207232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2631547620864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2577272136911 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9400596 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50450858916640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53023005409792 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60148108 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14798086 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60148108 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2584603888128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55607627300864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35390776 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24757330 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55607627300864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35390776 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24757330 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2584370207232 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55607375617024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53028131053551 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 Er+Ér.on_train_batch_end/allocation.large_pool.freed‚24757330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14798088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53027885032907 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58792592944128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚1629.599651977 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55607627300864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60121738 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56160734970368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2577043155627 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50450841877280 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2584603888128 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24198682 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56160758910464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35390776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24757332 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53023023412736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2584370207232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2631547620864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2577272136911 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9400596 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50450858916640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53023005409792 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60148108 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14798086 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273i»îä2‹- _wandbruntime‚1629  _runtime‚1629—¦üò[ŒÕ /loss/FRt2/detection_loss‚2.5652946221331754 0loss/SRqqt2/detection_loss‚2.737670307358106 .loss/FBt1/detection_loss‚2.564844325184822 /loss/FBt4/detection_loss‚1.3673384615918507 epoch‚1 0loss/FBt1/assignment_loss‚0.3571734693830024 2loss/FBt3/assignment_loss‚0.024979280445647116 /loss/FBt3/detection_loss‚1.3557069510221482 0loss/FRt1/assignment_loss‚3.7874158695340157 /loss/FRt2/assignment_loss‚4.386533121267955 0loss/FRt4/assignment_loss‚4.6880586097637815 1loss/SRqqt1/detection_loss‚1.9663957915407546 1loss/SRqqt3/detection_loss‚1.1091043350517475 0loss/SRqqt4/detection_loss‚2.707369810342789 trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚1.7927053405883464 2loss/SRqqt2/assignment_loss‚2.2234608891109624 0loss/FBt4/assignment_loss‚0.3355527584017314 %loss/total_loss‚50.52726970903032 _step‚106 0loss/FRt3/assignment_loss‚2.4474554468284953 .loss/FRt1/detection_loss‚2.803333694115281 /loss/FRt3/detection_loss‚1.8972390295203887 /loss/FBt2/detection_loss‚2.5962003977210433 2loss/SRqqt4/assignment_loss‚1.9243160717189312 -loss/FRt4/detection_loss‚3.17731910943985 % _timestamp‚1.7885306454478312e+09 _runtime‚1629.601160591 2loss/SRqqt3/assignment_loss‚1.4320789311046993 1loss/FBt2/assignment_loss‚0.27972308586079936¨r’‚_Ù epoch‚1 0loss/FBt1/assignment_loss‚0.3571734693830024 2loss/FBt3/assignment_loss‚0.024979280445647116 /loss/FBt3/detection_loss‚1.3557069510221482 0loss/FRt1/assignment_loss‚3.7874158695340157 /loss/FRt2/assignment_loss‚4.386533121267955 0loss/FRt4/assignment_loss‚4.6880586097637815 1loss/SRqqt1/detection_loss‚1.9663957915407546 1loss/SRqqt3/detection_loss‚1.1091043350517475 0loss/SRqqt4/detection_loss‚2.707369810342789 trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚1.7927053405883464 2loss/SRqqt2/assignment_loss‚2.2234608891109624 0loss/FBt4/assignment_loss‚0.3355527584017314 %loss/total_loss‚50.52726970903032 _step‚106 0loss/FRt3/assignment_loss‚2.4474554468284953 .loss/FRt1/detection_loss‚2.803333694115281 /loss/FRt3/detection_loss‚1.8972390295203887 /loss/FBt2/detection_loss‚2.5962003977210433 2loss/SRqqt4/assignment_loss‚1.9243160717189312 -loss/FRt4/detection_loss‚3.17731910943985 % _timestamp‚1.7885306454478312e+09 _runtime‚1629.601160591 2loss/SRqqt3/assignment_loss‚1.4320789311046993 1loss/FBt2/assignment_loss‚0.27972308586079936 /loss/FRt2/detection_loss‚2.5652946221331754 0loss/SRqqt2/detection_loss‚2.737670307358106 .loss/FBt1/detection_loss‚2.564844325184822 /loss/FBt4/detection_loss‚1.3673384615918507j>óÑY‘Ž‚(:† ÚŸëÔÇó˜ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2684 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åïíŸx‚(:í ÚŸëÔ–’„œ disk.sda1.out‚0 disk.sda.out‚ 51.93359375 disk.sda2.out‚0 network.sent‚ 2.043198e+06% disk./.usageGB‚1176.8171310424805 disk.drbd1260.in‚0 disk.sda.in‚0& disk.nvme0n1.out‚30888.44482421875 disk.sda3.in‚0 disk.sda3.out‚ 51.93359375) proc.memory.availableMB‚ 1927.18359375 disk.sda2.in‚0 disk.nvme0n1.in‚ 3801.3671875* proc.memory.percent‚18.432974815368652 cpu‚25.336544635622484 proc.cpu.threads‚17 network.recv‚583546) disk./.usagePercent‚78.97718507960236# disk.dm-0.out‚30888.44482421875 disk.sda1.in‚0 disk.drbd1260.out‚ 0.2265625$ memory_percent‚88.23740482330322# proc.memory.rssMB‚ 3020.05859375" disk.nvme0c0n1.in‚ 3801.3671875( disk.nvme0c0n1.out‚30888.44482421875 disk.dm-0.in‚ 3801.3671875 disk.dm-1.in‚0 disk.dm-1.out‚ 51.93359375!ë:Ÿ‘‚(:† éŸëÔçõÖ• gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0358 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÂzj‘‚(:ß éŸëԷ network.recv‚590743 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚88.24048042297363* proc.memory.percent‚18.432974815368652 cpu‚25.33473010485756) disk./.usagePercent‚78.97561012125173% disk./.usageGB‚1176.7936630249023# disk.dm-0.out‚31213.05029296875 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚31213.05029296875( proc.memory.availableMB‚ 1926.6796875( disk.nvme0c0n1.out‚31213.05029296875 disk.dm-0.in‚ 3922.77734375 disk.dm-1.out‚52.625 disk.drbd1260.out‚ 0.2265625 disk.sda.out‚52.625# proc.memory.rssMB‚ 3020.05859375 proc.cpu.threads‚17 network.sent‚ 2.151651e+06# disk.nvme0c0n1.in‚ 3922.77734375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 3922.77734375 disk.sda3.out‚52.625[> J’‚(:… øŸëÔóòË– gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9496 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ÚÑÉw“‚(:ì øŸëÔ­Ðà™ disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚88.23742866516113( proc.memory.availableMB‚ 1927.1796875 proc.cpu.threads‚17 network.sent‚ 2.15843e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 53.08984375 disk.drbd1260.out‚0.234375 disk.sda.out‚ 53.08984375! disk.nvme0n1.in‚ 3922.77734375& disk.nvme0n1.out‚31426.83837890625 disk.sda3.out‚ 53.08984375) disk./.usagePercent‚78.97718507960236# disk.nvme0c0n1.in‚ 3922.77734375# disk.dm-0.out‚31426.83837890625 disk.drbd1260.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 cpu‚25.33258962218465% disk./.usageGB‚1176.8171310424805( disk.nvme0c0n1.out‚31426.83837890625 disk.sda1.in‚0 disk.sda.in‚0 network.recv‚593959 disk.dm-0.in‚ 3922.77734375 disk.sda1.out‚0_¹kè‘”‚(:† ‡ ëÔ’ù©š gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚130.243' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0486 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂB"d•‚(:Ù ‡ ëÔõ¾Ò¥ disk.sda.in‚0 disk.sda3.out‚53.5 cpu‚25.330634370026598 network.sent‚ 2.16285e+06 disk.dm-1.in‚0 disk.dm-1.out‚53.5 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚31999.24853515625 disk.dm-0.in‚ 3922.77734375 disk.sda.out‚53.5& disk.nvme0n1.out‚31999.24853515625# disk.nvme0c0n1.in‚ 3922.77734375) proc.memory.availableMB‚ 1927.17578125# proc.memory.rssMB‚ 3020.05859375 network.recv‚598615) disk./.usagePercent‚78.97718635964004 disk.sda1.out‚0! disk.nvme0n1.in‚ 3922.77734375# disk.dm-0.out‚31999.24853515625$ memory_percent‚88.23745250701904* proc.memory.percent‚18.432974815368652 proc.cpu.threads‚17% disk./.usageGB‚1176.8171501159668 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.234375Å&ÎÍ2–- _wandbruntime‚1689  _runtime‚1689`,äÅÇ—Á _step‚107 & lr-AdamW/pg1‚0.0008162777993969447 & lr-AdamW/pg2‚0.0008162777993969447 trainer/global_step‚1349 % _timestamp‚1.7885307053317077e+09 _runtime‚1689.485560287€B6³Ë˜Å & lr-AdamW/pg2‚0.0008162777993969447 trainer/global_step‚1349 % _timestamp‚1.7885307053317077e+09 _runtime‚1689.485560287 _step‚107 & lr-AdamW/pg1‚0.0008162777993969447k ŠCv2™- _wandbruntime‚1689  _runtime‚1689u‡D{Æ'šÀO XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57628894780928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25652807 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25652807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52289846430416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58203273442816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57628894780928 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54948039225344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57628643110400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36702010 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15327386 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60932941015040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306171904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9739354 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2673022707149 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62328473 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2673251688225 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2729667572224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2680621888000 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2680855555584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58203249502720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25652805 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15327384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2729667572224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2680621888000 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2680855555584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58203249502720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25652805 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15327384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57628643110400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36702010 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15327386 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60932941015040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306171904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9739354 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2673022707149 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496l3ôI®2œ- _wandbruntime‚1690  _runtime‚1690âôž›Ï&ÉM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2731669780992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54987325262336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52327359748480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2674981196135 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚109 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62373440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57670144857088 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57670144857088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15338186 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52327376787840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62399810 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25084565 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682819594752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58244957412864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62399810 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚1690.521413931 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25671080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36728728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62373440 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25671082 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36728728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54987325262336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25671080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2675210177419ýDR(Ó&žÍM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52327376787840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62399810 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25084565 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682819594752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58244957412864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62399810 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚1690.521413931 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25671080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36728728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62373440 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25671082 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36728728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54987325262336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25671080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2675210177419 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2731669780992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54987325262336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52327359748480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2674981196135 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚109 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62373440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57670144857088 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57670144857088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15338186 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54166364301616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2771189728733 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 _step‚112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38013620 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15867486 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15867484 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10086734 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_trw¦ÿ Òain_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2779071262208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26566557 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2829604770304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59691160666624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56912341074944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56912323072000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60287471945216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59691160666624 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56912341074944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25954218 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38013620 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59691412337152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2770960747657 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59691412337152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2778837594624 _runtime‚ 1747.6788333 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26566555 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25954885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304074752 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64580175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38039962 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10087399 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63117076715520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496:>¯ÏÊ'¯ÄO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25954885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304074752 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64580175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38039962 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10087399 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63117076715520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54166364301616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2771189728733 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 _step‚112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38013620 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15867486 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15867484 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10086734 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2779071262208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26566557 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2829604770304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59691160666624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56912341074944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56912323072000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60287471945216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59691160666624 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56912341074944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25954218 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38013620 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59691412337152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2770960747657 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59691412337152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2778837594624 _runtime‚ 1747.6788333 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26566555 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0pp1Æ-2°- _wandbruntime‚1748  _runtime‚1748U¤’¨Ò&±ÌM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚113 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38066680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26584832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2780801620480 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1748.942743133 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64651512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63160453889024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56951609108992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64625142 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2772919236643 trainer/global_step‚1399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584830 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56976796856323 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781035301376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63160764241920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59732662413312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56951627111936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59732662413312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56977042876967 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54203894659040 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54203877619680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15878286 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25971704 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64651512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15878288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2831608326656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60329131975168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26584830 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¤Iž+=²ÐM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60329131975168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26584830 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚113 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38066680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26584832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2780801620480 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1748.942743133 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64651512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63160453889024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚56951609108992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64625142 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2772919236643 trainer/global_step‚1399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584830 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56976796856323 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781035301376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63160764241920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59732662413312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56951627111936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59732662413312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56977042876967 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54203894659040 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54203877619680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15878286 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25971704 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64651512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15878288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2831608326656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273qF_g2³- _wandbruntime‚1748  _runtime‚1748 =üåZ´Ô 1loss/SRqqt2/assignment_loss‚2.032714755304398 1loss/SRqqt3/detection_loss‚1.2352409843317789 /loss/FBt2/detection_loss‚2.7108623425165814 /loss/FRt2/detection_loss‚2.5199917309424458 /loss/FRt2/assignment_loss‚3.969623292193693 0loss/FBt1/assignment_loss‚0.5420001776076658 .loss/FRt4/detection_loss‚2.791349674974169 1loss/SRqqt2/detection_loss‚2.6480016381509843 .loss/FBt3/detection_loss‚1.076374624940482 &loss/total_loss‚49.030203815941185 epoch‚1 _runtime‚1748.944155441 /loss/FRt3/assignment_loss‚2.125208510206891 /loss/FRt4/assignment_loss‚4.840022044522422 1loss/FBt2/assignment_loss‚0.37061001754203365 3loss/FBt3/assignment_loss‚0.0017306140461046998 /loss/FBt4/detection_loss‚1.3710642989568265 trainer/global_step‚1399 _step‚114 2loss/SRqqt1/assignment_loss‚1.9235835847885985 .loss/FRt1/detection_loss‚2.450336068104475 1loss/SRqqt1/detection_loss‚2.2272060780148757 1loss/SRqqt4/detection_loss‚2.2226474092852686 /loss/FBt1/detection_loss‚2.3039055658238277 % _timestamp‚1.7885307647907405e+09 2loss/SRqqt4/assignment_loss‚2.0793336625060728 0loss/FBt4/assignment_loss‚0.2688404053590276 /loss/FRt1/assignment_loss‚4.356310106240786 1loss/SRqqt3/assignment_loss‚1.286395148778654 .loss/FRt3/detection_loss‚1.676851080803128+,hÌ^µØ /loss/FRt1/assignment_loss‚4.356310106240786 1loss/SRqqt3/assignment_loss‚1.286395148778654 .loss/FRt3/detection_loss‚1.676851080803128 1loss/SRqqt2/assignment_loss‚2.032714755304398 1loss/SRqqt3/detection_loss‚1.2352409843317789 /loss/FBt2/detection_loss‚2.7108623425165814 /loss/FRt2/detection_loss‚2.5199917309424458 /loss/FRt2/assignment_loss‚3.969623292193693 0loss/FBt1/assignment_loss‚0.5420001776076658 .loss/FRt4/detection_loss‚2.791349674974169 1loss/SRqqt2/detection_loss‚2.6480016381509843 .loss/FBt3/detection_loss‚1.076374624940482 &loss/total_loss‚49.030203815941185 epoch‚1 _runtime‚1748.944155441 /loss/FRt3/assignment_loss‚2.125208510206891 /loss/FRt4/assignment_loss‚4.840022044522422 1loss/FBt2/assignment_loss‚0.37061001754203365 3loss/FBt3/assignment_loss‚0.0017306140461046998 /loss/FBt4/detection_loss‚1.3710642989568265 trainer/global_step‚1399 _step‚114 2loss/SRqqt1/assignment_loss‚1.9235835847885985 .loss/FRt1/detection_loss‚2.450336068104475 1loss/SRqqt1/detection_loss‚2.2272060780148757 1loss/SRqqt4/detection_loss‚2.2226474092852686 /loss/FBt1/detection_loss‚2.3039055658238277 % _timestamp‚1.7885307647907405e+09 2loss/SRqqt4/assignment_loss‚2.0793336625060728 0loss/FBt4/assignment_loss‚0.2688404053590276r ‰B[œ¶‚(:‘ Ò ëÔ¶¦ï˜ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.765' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.352999999999998 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÅä’v·‚(:ë Ò ëÔ›‚à™& disk.nvme0n1.out‚33319.51611328125 disk.sda3.in‚0# disk.dm-0.out‚33319.51611328125 disk.sda1.out‚0$ memory_percent‚88.24095726013184$ disk./.usageGB‚1177.233741760254 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.24609375 disk.dm-1.in‚0# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 network.sent‚ 2.29109e+06 disk.nvme0n1.in‚ 4049.2578125( disk.nvme0c0n1.out‚33319.51611328125 disk.dm-1.out‚ 56.23046875 disk.sda1.in‚0( proc.memory.availableMB‚ 1926.6015625 cpu‚25.321041778635955 network.recv‚619496 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 56.23046875" disk.nvme0c0n1.in‚ 4049.2578125 disk.dm-0.in‚ 4049.2578125 proc.cpu.threads‚17) disk./.usagePercent‚79.00514417443182 disk.sda.out‚ 56.23046875 disk.sda2.in‚0÷nܸ‚(:… á ëÔÝË‹˜ gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1986 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ’ÁÜr¹‚(:ç á ëÔÐŒ“› network.recv‚625973% disk./.usageGB‚1177.2567710876465 disk.sda3.in‚0 disk.sda3.out‚ 56.66796875 disk.nvme0c0n1.in‚ 4191.21875 disk.sda1.in‚0) proc.memory.availableMB‚ 1926.83984375 disk.sda2.in‚0 disk.nvme0n1.in‚ 4191.21875& disk.nvme0n1.out‚33587.27001953125# disk.dm-0.out‚33587.27001953125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.24609375$ memory_percent‚88.23950290679932# proc.memory.rssMB‚ 3020.05859375 disk.sda2.out‚0( disk.nvme0c0n1.out‚33587.27001953125 disk.dm-0.in‚ 4191.21875 disk.dm-1.out‚ 56.66796875 disk.sda1.out‚0* proc.memory.percent‚18.432974815368652 cpu‚25.31916522154258 proc.cpu.threads‚17) disk./.usagePercent‚79.00668969191604 disk.sda.in‚0 disk.sda.out‚ 56.66796875 disk.dm-1.in‚0 network.sent‚ 2.399733e+06kÎÒýº‚(:… ð ëÔ¦ºö• gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žúpm»‚(:â ð ëÔøý˜# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652) disk./.usagePercent‚79.00680566332903% disk./.usageGB‚1177.2584991455078 disk.sda.in‚0 disk.sda.out‚ 57.26171875 disk.nvme0n1.in‚ 4191.21875 disk.sda3.in‚0 disk.sda3.out‚ 57.26171875# disk.dm-0.out‚34021.66650390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚34021.66650390625 disk.nvme0c0n1.in‚ 4191.21875( disk.nvme0c0n1.out‚34021.66650390625 disk.dm-0.in‚ 4191.21875 disk.dm-1.out‚ 57.26171875 disk.drbd1260.out‚0.25 cpu‚25.317560853949253 proc.cpu.threads‚17 network.sent‚ 2.404573e+06 network.recv‚629591 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚88.24102878570557) proc.memory.availableMB‚ 1926.58984375©ÖÖ‘¼‚(:† ÿ ëÔ¯¨¨˜ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.686' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3372 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*-~še½‚(:Ú ÿ ëÔùØõ disk.nvme0n1.in‚ 4191.21875 disk.sda3.out‚ 57.6015625' disk.nvme0c0n1.out‚34022.0244140625 disk.dm-0.in‚ 4191.21875 disk.drbd1260.out‚0.25% disk./.usageGB‚1177.2581520080566 disk.sda1.in‚0$ memory_percent‚90.32940864562988( proc.memory.availableMB‚ 1584.4296875* proc.memory.percent‚18.432974815368652 network.sent‚ 2.410786e+06 disk.sda.out‚ 57.6015625 disk.sda2.out‚0% disk.nvme0n1.out‚34022.0244140625 disk.dm-1.out‚ 57.6015625 disk.sda1.out‚0# proc.memory.rssMB‚ 3020.05859375 cpu‚25.31577865445565 disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 4191.21875" disk.dm-0.out‚34022.0244140625 disk.dm-1.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.recv‚634147) disk./.usagePercent‚79.00678236664342 disk.sda2.in‚0$JF2¾- _wandbruntime‚1806  _runtime‚1806h Ç¿Á _step‚115 & lr-AdamW/pg1‚0.0008156314487412975 & lr-AdamW/pg2‚0.0008156314487412975 trainer/global_step‚1449 % _timestamp‚1.7885308225173264e+09 _runtime‚1806.670798143»ûaÐËÀÅ & lr-AdamW/pg1‚0.0008156314487412975 & lr-AdamW/pg2‚0.0008156314487412975 trainer/global_step‚1449 % _timestamp‚1.7885308225173264e+09 _runtime‚1806.670798143 _step‚115s0ÿP€2Á- _wandbruntime‚1806  _runtime‚1806T›%ûÈ'ÂÂO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16407584 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58911763921621 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚1806.672067024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56042882172816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2868898788165 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66831877 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61753678222848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58876624921600 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39377914 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62371646507520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10433361 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62371670447616 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27480307 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65301138947584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26840945 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2929468499968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2929184171008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39351572 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58912009942057 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66858221 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944,Ÿ÷,:"ÃÆO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39351572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16407586 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10434041 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877053301248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58876642924544 _step‚116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2877053301248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚1449 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2877286968832 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885308225183759e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27480305 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58876624921600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2869127769241 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61753678222848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10433361 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62371670447616 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27480307 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65301138947584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26840945 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2929468499968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2929184171008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39351572 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58912009942057 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66858221 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16407584 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58911763921621 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚1806.672067024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56042882172816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2868898788165 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66831877 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61753678222848 H>DeviceStatsMonitor.on_train_batch_sta§f’rt/allocation.all.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58876624921600 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39377914 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62371646507520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2870857277151 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58915928961536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879251008000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66903214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61795179969536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39404632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2931182696448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61795179969536 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65344513174016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58951252768031 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26859154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61794928285696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62413354417664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2871086258435 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39404632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58915928961536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26858505 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58951498788675 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56080412530240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚117 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27498582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58915910958592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62413330477568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879017327104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66876844 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56080395490880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 $ _timestamp‚1.788530823476719e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16418388 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66903214 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0@¬ÛÝÕ&ÆÏM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2931182696448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61795179969536 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65344513174016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58951252768031 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26859154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61794928285696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62413354417664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2871086258435 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39404632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58915928961536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26858505 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58951498788675 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56080412530240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚117 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27498582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58915910958592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62413330477568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879017327104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66876844 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56080395490880 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 $ _timestamp‚1.788530823476719e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16418388 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66903214 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚1449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27498580 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚1807.631068145 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66876844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879017327104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2870857277151 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58915928961536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879251008000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66903214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61795179969536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39404632u2Ç- _wandbruntime‚1807  _runtime‚1807N@[ÈÕ 1loss/SRqqt2/assignment_loss‚2.271784190778379 /loss/FRt3/detection_loss‚1.4069903274796096 1loss/SRqqt4/detection_loss‚2.8902584456262135 % _timestamp‚1.7885308234786587e+09 /loss/FRt4/assignment_loss‚4.989822718832228 /loss/FRt4/detection_loss‚3.3646710216999054 .loss/FBt2/detection_loss‚2.702494477524477 /loss/FBt4/detection_loss‚1.3600523913455278 _runtime‚1807.631993659 1loss/SRqqt1/assignment_loss‚1.677921552459399 1loss/SRqqt2/detection_loss‚2.7456224582813404 .loss/FRt1/assignment_loss‚4.03612046145104 1loss/FBt2/assignment_loss‚0.26168307374786415 .loss/FRt1/detection_loss‚2.698163811419461 1loss/SRqqt3/detection_loss‚0.9728175001130217 %loss/total_loss‚50.27630881500026 /loss/FRt2/assignment_loss‚5.136260620185307 2loss/SRqqt3/assignment_loss‚1.1717158167844728 1loss/SRqqt1/detection_loss‚1.9139629854096307 trainer/global_step‚1449 2loss/SRqqt4/assignment_loss‚1.8270221608025687 0loss/FBt4/assignment_loss‚0.3205375717169303 1loss/FBt1/assignment_loss‚0.39994315371950506 4loss/FBt3/assignment_loss‚0.00012153731836406223 .loss/FBt1/detection_loss‚2.717590034008026 /loss/FBt3/detection_loss‚0.9976348313502967 epoch‚1 /loss/FRt3/assignment_loss‚1.369649548846555 /loss/FRt2/detection_loss‚3.0434681241001402 _step‚118:µT«_ÉÙ 1loss/FBt1/assignment_loss‚0.39994315371950506 4loss/FBt3/assignment_loss‚0.00012153731836406223 .loss/FBt1/detection_loss‚2.717590034008026 /loss/FBt3/detection_loss‚0.9976348313502967 epoch‚1 /loss/FRt3/assignment_loss‚1.369649548846555 /loss/FRt2/detection_loss‚3.0434681241001402 _step‚118 1loss/SRqqt2/assignment_loss‚2.271784190778379 /loss/FRt3/detection_loss‚1.4069903274796096 1loss/SRqqt4/detection_loss‚2.8902584456262135 % _timestamp‚1.7885308234786587e+09 /loss/FRt4/assignment_loss‚4.989822718832228 /loss/FRt4/detection_loss‚3.3646710216999054 .loss/FBt2/detection_loss‚2.702494477524477 /loss/FBt4/detection_loss‚1.3600523913455278 _runtime‚1807.631993659 1loss/SRqqt1/assignment_loss‚1.677921552459399 1loss/SRqqt2/detection_loss‚2.7456224582813404 .loss/FRt1/assignment_loss‚4.03612046145104 1loss/FBt2/assignment_loss‚0.26168307374786415 .loss/FRt1/detection_loss‚2.698163811419461 1loss/SRqqt3/detection_loss‚0.9728175001130217 %loss/total_loss‚50.27630881500026 /loss/FRt2/assignment_loss‚5.136260620185307 2loss/SRqqt3/assignment_loss‚1.1717158167844728 1loss/SRqqt1/detection_loss‚1.9139629854096307 trainer/global_step‚1449 2loss/SRqqt4/assignment_loss‚1.8270221608025687 0loss/FBt4/assignment_loss‚0.3205375717169303v>¥QœÊ‚(:‘ Ž¡ëÔµ´Ë– gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.308' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.061600000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò¡¹°jË‚(:ß Ž¡ëÔ“Ṙ disk.nvme0n1.in‚ 4191.21875) proc.memory.availableMB‚ 1584.39453125 proc.cpu.threads‚17 disk.dm-0.out‚34358.89453125 disk.dm-1.in‚0 disk.dm-1.out‚ 58.00390625 disk.sda2.in‚0 disk.sda3.out‚ 58.00390625* proc.memory.percent‚18.432974815368652 network.recv‚637548 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.25390625 disk.sda.out‚ 58.00390625 cpu‚25.313914183482126) disk./.usagePercent‚79.00163687121272% disk./.usageGB‚1177.1814804077148% disk.nvme0c0n1.out‚34358.89453125 disk.dm-0.in‚ 4191.21875# disk.nvme0n1.out‚34358.89453125 disk.sda3.in‚0$ memory_percent‚90.32962322235107# proc.memory.rssMB‚ 3020.05859375 network.sent‚ 2.415651e+06 disk.nvme0c0n1.in‚ 4191.21875 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.out‚0LAœ¤‘Ì‚(:† ¡ëÔ좣— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5542 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö,DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2975269007872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63816447449600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.cur}Y¡ã©rent‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40689524 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60840926771200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 _step‚120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69109923 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10781602 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3029263127040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40715866 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67485132077056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27728616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288523264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚1865.884823105 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63816447449600 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60840944774144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64455845009920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28394057 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63816195779072 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64455868950016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28394055 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67484819613696 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10780932 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3028974603776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60886465853765 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582Iü"ÇË'×ÅO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40689524 trainer/global_step‚1499 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2966836828673 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2975502675456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60840944774144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2975269007872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10780932 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3028974603776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60886465853765 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28394057 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2975502675456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312463360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28394055 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16947684 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60886219833329 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57919400044016 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2967065809749 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60840926771200 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63816195779072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27729288 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 $ _timestamp‚1.788530881730243e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69083579 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2975269007872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63816447449600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40689524 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60840926771200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 _step‚120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69109923 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10781602 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3029263127040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40715866 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67485132077056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27728616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288523264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚1865.884823105 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63816447449600 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60840944774144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64455845009920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28394057 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63816195779072 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64455868950016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28394055 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67484819613696 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60880212808192 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67528494173696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40742584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10788466 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67528808720896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69154916 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27746954 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40716216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16958488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 _step‚121 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60880230811136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57956913362080 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69128546 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚63857445841920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2977466714624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28412332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2969024298943 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27746273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16958486 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63857697525760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2977233033728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28412330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69154916 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28412330 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788530882814989e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3030965193728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64497528979968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚60925708679739 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608ôƒ€fÚÏM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16958488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 _step‚121 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60880230811136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57956913362080 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69128546 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚63857445841920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2977466714624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28412332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2969024298943 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27746273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16958486 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63857697525760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2977233033728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28412330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69154916 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28412330 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788530882814989e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3030965193728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64497528979968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚60925708679739 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60880212808192 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67528494173696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40742584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10788466 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67528808720896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69154916 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27746954 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40716216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712yt´®Ü2Û- _wandbruntime‚1866  _runtime‚18664¤SÜÍ /loss/FRt4/assignment_loss‚4.877824859619141 1loss/FBt2/assignment_loss‚0.21119100719933862 0loss/FBt4/assignment_loss‚0.2727689217482248 1loss/SRqqt4/detection_loss‚2.2514649514968577 _step‚122 1loss/SRqqt2/detection_loss‚2.3260772367939353 /loss/FBt4/detection_loss‚1.1849866554287611 .loss/FRt4/detection_loss‚3.002092864513397 %loss/total_loss‚48.63844780490915 trainer/global_step‚1499 0loss/SRqqt3/detection_loss‚1.084933384173465 epoch‚1 $ _timestamp‚1.788530882818926e+09 0loss/FRt3/assignment_loss‚1.6678512010309432 0loss/FBt1/assignment_loss‚0.3591319940230613 .loss/FRt1/detection_loss‚2.833557342290878 1loss/SRqqt1/detection_loss‚2.2000211295558185 -loss/FBt2/detection_loss‚2.40796622633934 /loss/FRt1/assignment_loss‚3.915461983680725 2loss/SRqqt1/assignment_loss‚1.7864152849083994 .loss/FRt2/detection_loss‚3.233147815517757 /loss/FBt3/detection_loss‚0.8258487683805552 /loss/FRt2/assignment_loss‚4.876426261404286 0loss/SRqqt4/assignment_loss‚1.84617030276702 .loss/FRt3/detection_loss‚1.418382432932655 .loss/FBt1/detection_loss‚2.389753415396339 _runtime‚1866.972358237 1loss/SRqqt2/assignment_loss‚2.329541322076693 2loss/SRqqt3/assignment_loss‚1.3211512015074154 2loss/FBt3/assignment_loss‚0.016281242124146642²BçTWÝÑ $ _timestamp‚1.788530882818926e+09 0loss/FRt3/assignment_loss‚1.6678512010309432 0loss/FBt1/assignment_loss‚0.3591319940230613 .loss/FRt1/detection_loss‚2.833557342290878 1loss/SRqqt1/detection_loss‚2.2000211295558185 -loss/FBt2/detection_loss‚2.40796622633934 /loss/FRt1/assignment_loss‚3.915461983680725 2loss/SRqqt1/assignment_loss‚1.7864152849083994 .loss/FRt2/detection_loss‚3.233147815517757 /loss/FBt3/detection_loss‚0.8258487683805552 /loss/FRt2/assignment_loss‚4.876426261404286 0loss/SRqqt4/assignment_loss‚1.84617030276702 .loss/FRt3/detection_loss‚1.418382432932655 .loss/FBt1/detection_loss‚2.389753415396339 _runtime‚1866.972358237 1loss/SRqqt2/assignment_loss‚2.329541322076693 2loss/SRqqt3/assignment_loss‚1.3211512015074154 2loss/FBt3/assignment_loss‚0.016281242124146642 /loss/FRt4/assignment_loss‚4.877824859619141 1loss/FBt2/assignment_loss‚0.21119100719933862 0loss/FBt4/assignment_loss‚0.2727689217482248 1loss/SRqqt4/detection_loss‚2.2514649514968577 _step‚122 1loss/SRqqt2/detection_loss‚2.3260772367939353 /loss/FBt4/detection_loss‚1.1849866554287611 .loss/FRt4/detection_loss‚3.002092864513397 %loss/total_loss‚48.63844780490915 trainer/global_step‚1499 0loss/SRqqt3/detection_loss‚1.084933384173465 epoch‚1zF¼®éœÞ‚(:‘ Ê¡ëÔÌǸ˜ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.168' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.233600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô8xß‚(:í Ê¡ëÔŠ®¹™' disk.nvme0c0n1.out‚35643.2666015625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.26171875 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0* proc.memory.percent‚18.432974815368652 network.sent‚ 2.54015e+06 disk.dm-1.in‚0 disk.sda.out‚ 59.77734375% disk.nvme0n1.out‚35643.2666015625 disk.sda3.out‚ 59.77734375) proc.memory.availableMB‚ 1584.04296875# proc.memory.rssMB‚ 3020.05859375 network.recv‚656223% disk./.usageGB‚1177.3194541931152 disk.dm-0.in‚ 4299.77734375 disk.sda2.in‚0! disk.nvme0n1.in‚ 4299.77734375 cpu‚25.307432224995885) disk./.usagePercent‚79.01089640771866# disk.nvme0c0n1.in‚ 4299.77734375" disk.dm-0.out‚35643.2666015625 disk.dm-1.out‚ 59.77734375 disk.sda1.in‚0$ memory_percent‚90.33176898956299 proc.cpu.threads‚17ˆÊlX‘à‚(:† Ù¡ëÔ£¯«— gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6662 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ¼]>xá‚(:í Ù¡ëÔ¦¶¾›# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 cpu‚25.305761624229422 network.recv‚663361 disk.sda3.out‚ 60.41796875# disk.nvme0c0n1.in‚ 4299.77734375 proc.cpu.threads‚17( disk./.usagePercent‚79.0151008194535 disk.sda.in‚0! disk.nvme0n1.in‚ 4299.77734375% disk.nvme0n1.out‚35791.8759765625 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0% disk./.usageGB‚1177.3821029663086 disk.sda.out‚ 60.41796875 disk.sda2.out‚0' disk.nvme0c0n1.out‚35791.8759765625 disk.dm-0.in‚ 4299.77734375" disk.dm-0.out‚35791.8759765625 disk.drbd1260.out‚ 0.26171875$ memory_percent‚90.33029079437256 network.sent‚ 2.649085e+06 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 60.41796875) proc.memory.availableMB‚ 1584.28515625íeבâ‚(:† è¡ëÔ€—å— gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8716 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ynÆÕWã‚(:Ì è¡ëÔÑÑí˜ disk.sda3.out‚61$ memory_percent‚90.32881259918213) proc.memory.availableMB‚ 1584.52734375 cpu‚25.30437140431885$ disk./.usageGB‚1177.432674407959 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0 network.sent‚ 2.654151e+06! disk.nvme0c0n1.in‚ 4435.359375 disk.dm-0.in‚ 4435.359375 disk.drbd1260.out‚0.265625 disk.sda.out‚61 disk.sda2.out‚0# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 network.recv‚666574) disk./.usagePercent‚79.01849471133492# disk.dm-0.out‚36164.91845703125 disk.sda1.in‚0& disk.nvme0n1.out‚36164.91845703125 disk.sda3.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚36164.91845703125 disk.dm-1.out‚61 disk.nvme0n1.in‚ 4435.359375~!.^‘ä‚(:† ÷¡ëÔËü¢˜ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Åe:qå‚(:æ ÷¡ëÔŽ°¬š disk.drbd1260.out‚0.265625# proc.memory.rssMB‚ 3020.05859375) disk./.usagePercent‚79.02767130139976% disk./.usageGB‚1177.5694122314453 disk.sda3.out‚ 61.41796875 disk.dm-0.in‚ 4435.359375# disk.dm-0.out‚36550.49267578125 disk.sda1.in‚0 disk.sda.in‚0# memory_percent‚90.3303861618042 network.sent‚ 2.66033e+06 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 61.41796875* proc.memory.percent‚18.432974815368652 cpu‚25.30263665445912 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4435.359375& disk.nvme0n1.out‚36550.49267578125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 61.41796875) proc.memory.availableMB‚ 1584.26953125 network.recv‚671348 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 4435.359375( disk.nvme0c0n1.out‚36550.49267578125 disk.sda1.out‚0xí,°2æ- _wandbruntime‚1926  _runtime‚19262“WýÇçÁ & lr-AdamW/pg1‚0.0008146455714377204 & lr-AdamW/pg2‚0.0008146455714377204 trainer/global_step‚1549 % _timestamp‚1.7885309421858518e+09 _runtime‚1926.339480513 _step‚123üÛH©ËèÅ _step‚123 & lr-AdamW/pg1‚0.0008146455714377204 & lr-AdamW/pg2‚0.0008146455714377204 trainer/global_step‚1549 % _timestamp‚1.7885309421858518e+09 _runtime‚1926.339480513{A¡ 2é- _wandbruntime‚1926  _runtime‚1926L;t-È'êÂO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3065003850257 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59795900875856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3129077429248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65878965005824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65878713335296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71335281 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42053818 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1549 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3073718382080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11129789 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71361625 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17487786 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11129115 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59795917915216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62805246623744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3073484714496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42027476 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65878713335296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚1926.341295986 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3073484714496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62860921765473 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3128784711680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42053818 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69668828224000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62860675745037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17487784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885309421872673e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62805246623744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66540043512320 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62805228620800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42027476 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69669144881664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65878965005824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28616899 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42027476 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69669144881664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65878965005824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28616899 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3065003850257 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59795900875856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3129077429248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65878965005824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65878713335296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71335281 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42053818 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1549 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3073718382080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11129789 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71361625 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17487786 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11129115 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59795917915216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62805246623744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3073484714496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42027476 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65878713335296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚1926.341295986 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3073484714496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 BDeviceStòÃûZatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62860921765473 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496|ÑÓjÆ2ì- _wandbruntime‚1927  _runtime‚1927V`Ÿ>Ñ&íËM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1927.643092221 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3075682421248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3075448740352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3130780617728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65920215081984 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71380248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3131069127680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29326082 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59833431233280 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65920215081984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71406618 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71380248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1549 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28635396 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59833448272640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62844532660736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3075448740352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66581751422464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11136132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65919963398144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11136808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62844514657792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42054168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62844514657792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66581727482368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62900410612091 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3066962339451 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28634718« ÿÕÕ&îÏM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66581751422464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11136132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65919963398144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11136808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62844514657792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42054168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62844514657792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66581727482368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62900410612091 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3066962339451 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28634718 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1927.643092221 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3075682421248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3075448740352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3130780617728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65920215081984 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71380248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3131069127680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29326082 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59833431233280 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65920215081984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71406618 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71380248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1549 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28635396 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59833448272640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62844532660736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3075448740352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120}¥¤óŒ2ï- _wandbruntime‚1927  _runtime‚1927eä¿UðÏ /loss/FBt2/detection_loss‚2.3550705345613614 /loss/FBt4/detection_loss‚1.4381219882615512 1loss/SRqqt3/assignment_loss‚1.081980699804007 .loss/FRt1/detection_loss‚2.292431132590517 -loss/FRt3/detection_loss‚1.53105436557347 1loss/SRqqt4/detection_loss‚2.2498145681439023 /loss/FRt4/assignment_loss‚4.396324072033167 0loss/SRqqt2/detection_loss‚2.940509106801904 /loss/FRt2/detection_loss‚2.9755727159125462 1loss/FBt1/assignment_loss‚0.33290391899364685 /loss/FBt2/assignment_loss‚0.552617140195902 1loss/FBt4/assignment_loss‚0.19404392946159704 1loss/SRqqt3/detection_loss‚1.0255316906330878 &loss/total_loss‚48.669891506595036 _step‚126 .loss/FBt3/detection_loss‚1.062926894815808 epoch‚1 % _timestamp‚1.7885309434911594e+09 0loss/FRt1/assignment_loss‚3.9349993467330933 1loss/SRqqt2/assignment_loss‚2.010810346869023 trainer/global_step‚1549 _runtime‚1927.644573685 .loss/FRt2/assignment_loss‚5.66985206093107 .loss/FBt1/detection_loss‚2.258839262383325 2loss/SRqqt4/assignment_loss‚1.8745710208560482 .loss/FRt4/detection_loss‚2.889262594282627 0loss/FRt3/assignment_loss‚1.9280326778126746 1loss/SRqqt1/assignment_loss‚1.581571711187667 1loss/FBt3/assignment_loss‚0.06948854304639761 1loss/SRqqt1/detection_loss‚2.0235611847106445Dr;BYñÓ /loss/FBt2/detection_loss‚2.3550705345613614 /loss/FBt4/detection_loss‚1.4381219882615512 1loss/SRqqt3/assignment_loss‚1.081980699804007 .loss/FRt1/detection_loss‚2.292431132590517 -loss/FRt3/detection_loss‚1.53105436557347 1loss/SRqqt4/detection_loss‚2.2498145681439023 /loss/FRt4/assignment_loss‚4.396324072033167 0loss/SRqqt2/detection_loss‚2.940509106801904 /loss/FRt2/detection_loss‚2.9755727159125462 1loss/FBt1/assignment_loss‚0.33290391899364685 /loss/FBt2/assignment_loss‚0.552617140195902 1loss/FBt4/assignment_loss‚0.19404392946159704 1loss/SRqqt3/detection_loss‚1.0255316906330878 &loss/total_loss‚48.669891506595036 _step‚126 .loss/FBt3/detection_loss‚1.062926894815808 epoch‚1 % _timestamp‚1.7885309434911594e+09 0loss/FRt1/assignment_loss‚3.9349993467330933 1loss/SRqqt2/assignment_loss‚2.010810346869023 trainer/global_step‚1549 _runtime‚1927.644573685 .loss/FRt2/assignment_loss‚5.66985206093107 .loss/FBt1/detection_loss‚2.258839262383325 2loss/SRqqt4/assignment_loss‚1.8745710208560482 .loss/FRt4/detection_loss‚2.889262594282627 0loss/FRt3/assignment_loss‚1.9280326778126746 1loss/SRqqt1/assignment_loss‚1.581571711187667 1loss/FBt3/assignment_loss‚0.06948854304639761 1loss/SRqqt1/detection_loss‚2.0235611847106445~ ­ò‚(:„ †¢ëÔÀ«‰ž gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.83' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.966 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•öró‚(:ç †¢ëÔ¥œ³ % disk.nvme0n1.out‚36854.7353515625 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 4435.359375" disk.dm-0.out‚36854.7353515625 network.recv‚675039% disk./.usageGB‚1177.5694961547852 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4435.359375 disk.dm-0.in‚ 4435.359375( proc.memory.availableMB‚ 1583.9921875# proc.memory.rssMB‚ 3020.05859375) disk./.usagePercent‚79.02767693356552 disk.sda.out‚ 61.92578125 disk.sda3.out‚ 61.92578125$ memory_percent‚90.33207893371582* proc.memory.percent‚18.432974815368652 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 61.92578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.26953125' disk.nvme0c0n1.out‚36854.7353515625 cpu‚25.301316432968452 network.sent‚ 2.665509e+06 disk.sda.in‚0 disk.sda2.in‚0%å뇑ô‚(:† •¢ëÔÅÒ̘ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0026 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á…Xbrõ‚(:ç •¢ëÔ» ¾› disk.sda2.in‚0! disk.nvme0c0n1.in‚ 4435.359375' disk.nvme0c0n1.out‚37227.4033203125 disk.dm-0.in‚ 4435.359375 disk.drbd1260.in‚0# memory_percent‚90.3306245803833% disk./.usageGB‚1177.8194770812988 disk.nvme0n1.in‚ 4435.359375% disk.nvme0n1.out‚37227.4033203125 disk.sda3.out‚ 62.41015625 disk.dm-1.in‚0 disk.dm-1.out‚ 62.41015625 disk.drbd1260.out‚ 0.26953125# proc.memory.rssMB‚ 3020.05859375 disk.sda.out‚ 62.41015625 disk.sda1.in‚0) proc.memory.availableMB‚ 1584.23046875 cpu‚25.299672782852774) disk./.usagePercent‚79.04445336329181 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0" disk.dm-0.out‚37227.4033203125 disk.sda1.out‚0* proc.memory.percent‚18.432974815368652 proc.cpu.threads‚17 network.sent‚ 2.773945e+06 network.recv‚681641ŸŠlÑœö‚(:‘ ¤¢ëÔ¨œÌ˜ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.871' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.374200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úççl÷‚(:á ¤¢ëÔ—ª‡› disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 37541.2265625 disk.dm-0.in‚ 4575.98828125# proc.memory.rssMB‚ 3020.05859375 disk.dm-1.in‚0 disk.dm-1.out‚ 63.0078125) proc.memory.availableMB‚ 1584.48046875 cpu‚25.298185243687637 proc.cpu.threads‚17" disk.nvme0n1.out‚ 37541.2265625 disk.dm-0.out‚ 37541.2265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2734375* proc.memory.percent‚18.432974815368652 network.sent‚ 2.780775e+06 network.recv‚684892) disk./.usagePercent‚79.00743928197629! disk.nvme0n1.in‚ 4575.98828125 disk.sda3.in‚0 disk.sda3.out‚ 63.0078125# disk.nvme0c0n1.in‚ 4575.98828125$ memory_percent‚90.32909870147705% disk./.usageGB‚1177.2679405212402 disk.sda.in‚0 disk.sda.out‚ 63.0078125 disk.sda2.in‚0W(¿éø‚(:… ³¢ëÔÕ¤• gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚96.273' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2546 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞÈ‚ƒbù‚(:× ³¢ëԮ˻—$ memory_percent‚90.33222198486328& proc.memory.availableMB‚ 1583.96875 proc.cpu.threads‚17) disk./.usagePercent‚79.00744209805916 disk.nvme0n1.out‚ 37676.8125# disk.nvme0c0n1.in‚ 4575.98828125! disk.nvme0c0n1.out‚ 37676.8125 disk.dm-1.out‚ 63.35546875 disk.sda1.in‚0 disk.sda2.out‚0 network.recv‚689558 disk.sda3.in‚0 disk.dm-0.in‚ 4575.98828125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.2734375 disk.sda.in‚0# proc.memory.rssMB‚ 3020.05859375 network.sent‚ 2.785083e+06% disk./.usageGB‚1177.2679824829102 disk.dm-0.out‚ 37676.8125 disk.sda.out‚ 63.35546875* proc.memory.percent‚18.432974815368652 cpu‚25.29692885449191! disk.nvme0n1.in‚ 4575.98828125 disk.sda3.out‚ 63.35546875 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0µžÆ2ú- _wandbruntime‚1984  _runtime‚1984’^>LÇûÁ & lr-AdamW/pg2‚0.0008140255560607075 trainer/global_step‚1599 % _timestamp‚1.7885310008280783e+09 _runtime‚1984.981608896 _step‚127 & lr-AdamW/pg1‚0.0008140255560607075^bÆ;ËüÅ & lr-AdamW/pg1‚0.0008140255560607075 & lr-AdamW/pg2‚0.0008140255560607075 trainer/global_step‚1599 % _timestamp‚1.7885310008280783e+09 _runtime‚1984.981608896 _step‚127NÜôÜ2ý- _wandbruntime‚1984  _runtime‚1984»Íÿn þÂO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67941230891520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3171934088704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18027884 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _step‚128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30221555 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11479716 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3228518680064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3162941890765 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67941482562048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_ÊÃœ_Zbytes.large_pool.freed‚68624242014720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64769530470400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 % _timestamp‚1.7885310008295078e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1599 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1984.983707283 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64769548473344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11479009 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71853075255296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64835131656745 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3171934088704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30221555 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64769548473344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43365428 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61672435786416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3162712909689 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43391770 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73613327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30221557 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43365428 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64835377677181 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67941482562048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71852760694784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18027886 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43391770 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3171700421120 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73586983 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3171700421120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3228809300480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67941230891520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64769530470400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0­šŠÍ'ÿÇO trainer/global_step‚1599 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1984.983707283 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64769548473344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11479009 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71853075255296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64835131656745 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3171934088704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30221555 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64769548473344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43365428 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61672435786416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3162712909689 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43391770 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73613327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30221557 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43365428 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64835377677181 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67941482562048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71852760694784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18027886 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43391770 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3171700421120 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73586983 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3171700421120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3228809300480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67941230891520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64769530470400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67941230891520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3171934088704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18027884 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _step‚128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30221555 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11479716 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3228518680064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3162941890765 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67941482562048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68624242014720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64769530470400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 % _timestamp‚1.7885310008295078e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0€AÊÆ2€- _wandbruntime‚1986  _runtime‚1986E •Ò&ÌM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43392120 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30239832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18038688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11486392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43418488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71896754171904 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71896433333248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3173664446976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73658320 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61709949104480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64808816507392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3164671398675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30239832 _step‚129 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11485717 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68665949924864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73631950 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18038686 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885310022174554e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30239830 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67982732638208 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64808834510336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43392120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61709966143840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29525080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3164900379959 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚1986.372431784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29524403 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚1599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30239830 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3173898127872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3173898127872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3230804247040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67982732638208 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3173664446976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43418488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73658320 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769+Õ*²‚ÑM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73658320 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43392120 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30239832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18038688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11486392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43418488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71896754171904 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71896433333248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3173664446976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73658320 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61709949104480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64808816507392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3164671398675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30239832 _step‚129 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11485717 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68665949924864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73631950 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18038686 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885310022174554e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30239830 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67982732638208 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64808834510336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43392120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61709966143840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29525080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3164900379959 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚1986.372431784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29524403 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚1599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30239830 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3173898127872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3173898127872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3230804247040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67982732638208 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3173664446976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43418488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚05½Œ2ƒ- _wandbruntime‚1986  _runtime‚1986 –ÜšY„Ó 0loss/FBt4/assignment_loss‚0.2815213185228026 % _timestamp‚1.7885310022224805e+09 2loss/SRqqt1/assignment_loss‚1.5130100660728958 0loss/FBt1/assignment_loss‚0.6185770846308775 0loss/FBt3/assignment_loss‚0.0461025153301049 .loss/FBt1/detection_loss‚2.510863964046751 /loss/FBt3/detection_loss‚1.0061360453255475 trainer/global_step‚1599 /loss/FBt4/detection_loss‚1.5935919880111877 &loss/total_loss‚49.560933547202794 /loss/FRt1/detection_loss‚2.3338746792740293 .loss/FRt4/detection_loss‚3.525837080819266 1loss/SRqqt2/detection_loss‚2.7935880814727985 1loss/SRqqt3/detection_loss‚1.0409004402533173 epoch‚1 /loss/FRt2/assignment_loss‚4.416791154788091 0loss/FRt3/assignment_loss‚1.9442076903013956 /loss/FRt2/detection_loss‚3.1474020068462076 1loss/SRqqt1/detection_loss‚1.6911798648114473 2loss/SRqqt2/assignment_loss‚2.4786161968582556 2loss/SRqqt4/assignment_loss‚2.0600667867971505 0loss/SRqqt4/detection_loss‚2.503031831720601 _step‚130 .loss/FRt1/assignment_loss‚4.53751675453451 1loss/FBt2/assignment_loss‚0.34146747941678557 /loss/FBt2/detection_loss‚2.2156084425309124 /loss/FRt4/assignment_loss‚4.252862918944586 2loss/SRqqt3/assignment_loss‚1.1680453992635011 .loss/FRt3/detection_loss‚1.540133756629768 _runtime‚1986.375947622O¿=%^…Ø /loss/FRt1/detection_loss‚2.3338746792740293 .loss/FRt4/detection_loss‚3.525837080819266 1loss/SRqqt2/detection_loss‚2.7935880814727985 1loss/SRqqt3/detection_loss‚1.0409004402533173 epoch‚1 /loss/FRt2/assignment_loss‚4.416791154788091 0loss/FRt3/assignment_loss‚1.9442076903013956 /loss/FRt2/detection_loss‚3.1474020068462076 1loss/SRqqt1/detection_loss‚1.6911798648114473 2loss/SRqqt2/assignment_loss‚2.4786161968582556 2loss/SRqqt4/assignment_loss‚2.0600667867971505 0loss/SRqqt4/detection_loss‚2.503031831720601 _step‚130 .loss/FRt1/assignment_loss‚4.53751675453451 1loss/FBt2/assignment_loss‚0.34146747941678557 /loss/FBt2/detection_loss‚2.2156084425309124 /loss/FRt4/assignment_loss‚4.252862918944586 2loss/SRqqt3/assignment_loss‚1.1680453992635011 .loss/FRt3/detection_loss‚1.540133756629768 _runtime‚1986.375947622 0loss/FBt4/assignment_loss‚0.2815213185228026 % _timestamp‚1.7885310022224805e+09 2loss/SRqqt1/assignment_loss‚1.5130100660728958 0loss/FBt1/assignment_loss‚0.6185770846308775 0loss/FBt3/assignment_loss‚0.0461025153301049 .loss/FBt1/detection_loss‚2.510863964046751 /loss/FBt3/detection_loss‚1.0061360453255475 trainer/global_step‚1599 /loss/FBt4/detection_loss‚1.5935919880111877 &loss/total_loss‚49.560933547202794‚ñ¹¥Ž†‚(:ƒ ¢ëÔ³¿á™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vóÑöv‡‚(:ë ¢ëÔÇÛœ disk.sda1.out‚0# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 network.recv‚692777) disk./.usagePercent‚79.00743953798381$ disk.nvme0n1.out‚37942.751953125 disk.dm-1.out‚ 63.76171875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 63.76171875! disk.dm-0.out‚37942.751953125 disk.drbd1260.out‚ 0.27734375$ memory_percent‚90.33267498016357 cpu‚25.295339295310924 network.sent‚ 2.789953e+06 disk.sda.out‚ 63.76171875! disk.nvme0n1.in‚ 4575.98828125# disk.nvme0c0n1.in‚ 4575.98828125& disk.nvme0c0n1.out‚37942.751953125 disk.dm-1.in‚0 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1583.89453125% disk./.usageGB‚1177.2679443359375 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 4575.98828125 disk.sda1.in‚0®ç²ï‘ˆ‚(:† Ñ¢ëÔûã«— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚139.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.8238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÔ+ði‰‚(:Þ Ñ¢ëÔ¡ªú™ disk.dm-1.in‚0 disk.dm-1.out‚64.1875 disk.sda.in‚0 disk.sda.out‚64.1875 disk.sda3.out‚64.1875 network.recv‚699433! disk.dm-0.out‚38291.439453125! disk.nvme0n1.in‚ 4575.98828125# memory_percent‚90.3327226638794) proc.memory.availableMB‚ 1583.88671875* proc.memory.percent‚18.432974815368652 cpu‚25.293855645605202 proc.cpu.threads‚17 network.sent‚ 2.898423e+06) disk./.usagePercent‚79.00743953798381 disk.dm-0.in‚ 4575.98828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚1177.2679443359375 disk.drbd1260.out‚ 0.27734375$ disk.nvme0n1.out‚38291.439453125 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 4575.98828125& disk.nvme0c0n1.out‚38291.439453125# proc.memory.rssMB‚ 3020.05859375ÂÇIŠ‚(:„ à¢ëÔ—ͳ– gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.735' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.947 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U.ø˜u‹‚(:ê à¢ëÔ½Ïï› disk.sda3.in‚0$ memory_percent‚90.33119678497314) proc.memory.availableMB‚ 1584.13671875# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 network.recv‚701872 disk.sda3.out‚ 64.80859375 disk.dm-1.in‚0 disk.dm-1.out‚ 64.80859375 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.29263875437798 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 64.80859375 disk.sda2.out‚0 disk.dm-0.in‚ 4709.39453125" disk.dm-0.out‚38888.1435546875 disk.drbd1260.in‚0 network.sent‚ 2.903252e+06) disk./.usagePercent‚79.00743953798381 disk.sda2.in‚0! disk.nvme0n1.in‚ 4709.39453125# disk.nvme0c0n1.in‚ 4709.39453125' disk.nvme0c0n1.out‚38888.1435546875 disk.drbd1260.out‚0.28125% disk./.usageGB‚1177.2679443359375% disk.nvme0n1.out‚38888.1435546875§¨Ý^œŒ‚(:‘ ï¢ëԻƬ› gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚129.194' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.838799999999996 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áû(¯r‚(:ç ï¢ëÔ£ÉÁ£' disk.nvme0c0n1.out‚39101.3818359375 network.sent‚ 2.90854e+06 network.recv‚706239( disk./.usagePercent‚79.0074408180215 disk.sda.in‚0 disk.sda.out‚ 65.27734375 disk.sda2.in‚0 disk.sda3.in‚0" disk.dm-0.out‚39101.3818359375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.28125$ memory_percent‚90.33269882202148% disk./.usageGB‚1177.2679634094238 disk.sda2.out‚0! disk.nvme0n1.in‚ 4709.39453125 disk.dm-0.in‚ 4709.39453125 disk.dm-1.out‚ 65.27734375 disk.sda1.in‚0' proc.memory.availableMB‚ 1583.890625 cpu‚25.291174349208497 disk.sda3.out‚ 65.27734375# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 proc.cpu.threads‚17% disk.nvme0n1.out‚39101.3818359375# disk.nvme0c0n1.in‚ 4709.39453125zÀu÷2Ž- _wandbruntime‚2043  _runtime‚2043|´ò‰ÇÁ _step‚131 & lr-AdamW/pg1‚0.0008133209892648064 & lr-AdamW/pg2‚0.0008133209892648064 trainer/global_step‚1649 % _timestamp‚1.7885310593629498e+09 _runtime‚2043.516683352’g‡`ÌÆ & lr-AdamW/pg1‚0.0008133209892648064 & lr-AdamW/pg2‚0.0008133209892648064 trainer/global_step‚1649 % _timestamp‚1.7885310593629498e+09 _runtime‚2043.516683352 _step‚131ƒâ°GÔ2‘- _wandbruntime‚2043  _runtime‚2043÷[L_È'’ÂO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75838685 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70708440517120 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31135307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3260650950197 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885310593644128e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66809587568453 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44703380 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70004000118272 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70003748447744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3328592441344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63548936618256 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44729722 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30392008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66733832320000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66733832320000 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31135305 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66733850322944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44703380 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74037056898560 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74036736046592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3270149795328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3328295529472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚2043.518526336 _step‚132 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31135305 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63548953657616 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18567986 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3269916127744 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30392719 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70004000118272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66733850322944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BÙ¥&“ÇO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75865029 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11824024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3260879931273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18567984 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3270149795328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31135307 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11824733 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296911872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44703380 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70004000118272 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70003748447744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3328592441344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63548936618256 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44729722 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30392008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66733832320000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66733832320000 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31135305 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66733850322944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44703380 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74037056898560 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74036736046592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3270149795328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3328295529472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚2043.518526336 _step‚132 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31135305 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63548953657616 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18567986 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3269916127744 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30392719 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70004000118272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66733850322944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885310593644128e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66809587568453 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31153580 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18578786 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66849322435507 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3262609439183 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70045250194432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70750148427264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3271880153600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11831975 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3272113834496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74080420409856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885310605500553e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66773118356992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66773136359936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66849076414863 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75910022 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75910022 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30410763 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30410081 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _step‚133 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75883652 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44756440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75883652 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75910022 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75910022 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30410763 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30410081 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _step‚133 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75883652 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44756440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75883652 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31153580 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18578786 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66849322435507 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3262609439183 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70045250194432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70750148427264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3271880153600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11831975 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3272113834496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74080420409856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885310605500553e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66773118356992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66773136359936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0…:mU>2—- _wandbruntime‚2044  _runtime‚2044]é©]˜× .loss/FRt4/assignment_loss‚5.23616737978799 2loss/SRqqt3/assignment_loss‚1.5551757748154076 1loss/FBt4/assignment_loss‚0.16273640932941516 .loss/FRt4/detection_loss‚3.281151328768049 /loss/FBt4/detection_loss‚1.3301123347935693 /loss/FRt2/detection_loss‚3.3094631948254327 /loss/FBt1/detection_loss‚2.4763091146945952 trainer/global_step‚1649 $ _timestamp‚1.788531060552585e+09 %loss/total_loss‚51.22304603130313 /loss/FRt2/assignment_loss‚4.474375638094815 0loss/FRt3/assignment_loss‚1.8080305919718387 2loss/SRqqt1/assignment_loss‚1.7608153040831287 /loss/FBt3/detection_loss‚0.9030674656232198 epoch‚1 2loss/SRqqt4/assignment_loss‚1.5635967861723017 /loss/FRt3/detection_loss‚1.7615642090564343 /loss/FBt2/detection_loss‚2.5078555941581726 2loss/SRqqt2/assignment_loss‚1.4314378222916275 0loss/FBt1/assignment_loss‚0.7239977958275631 0loss/FBt2/assignment_loss‚0.3636319246075648 1loss/SRqqt4/detection_loss‚2.8745120697551303 _runtime‚2044.706148672 /loss/FRt1/assignment_loss‚4.699330190132404 /loss/FRt1/detection_loss‚3.0975190606610528 1loss/SRqqt2/detection_loss‚2.8486987724900246 3loss/FBt3/assignment_loss‚0.0045600150941462895 1loss/SRqqt1/detection_loss‚2.0822588748609028 1loss/SRqqt3/detection_loss‚0.9666783794083379 _step‚134¹ˆ"ÿb™Ü /loss/FRt2/assignment_loss‚4.474375638094815 0loss/FRt3/assignment_loss‚1.8080305919718387 2loss/SRqqt1/assignment_loss‚1.7608153040831287 /loss/FBt3/detection_loss‚0.9030674656232198 epoch‚1 2loss/SRqqt4/assignment_loss‚1.5635967861723017 /loss/FRt3/detection_loss‚1.7615642090564343 /loss/FBt2/detection_loss‚2.5078555941581726 2loss/SRqqt2/assignment_loss‚1.4314378222916275 0loss/FBt1/assignment_loss‚0.7239977958275631 0loss/FBt2/assignment_loss‚0.3636319246075648 1loss/SRqqt4/detection_loss‚2.8745120697551303 _runtime‚2044.706148672 /loss/FRt1/assignment_loss‚4.699330190132404 /loss/FRt1/detection_loss‚3.0975190606610528 1loss/SRqqt2/detection_loss‚2.8486987724900246 3loss/FBt3/assignment_loss‚0.0045600150941462895 1loss/SRqqt1/detection_loss‚2.0822588748609028 1loss/SRqqt3/detection_loss‚0.9666783794083379 _step‚134 .loss/FRt4/assignment_loss‚5.23616737978799 2loss/SRqqt3/assignment_loss‚1.5551757748154076 1loss/FBt4/assignment_loss‚0.16273640932941516 .loss/FRt4/detection_loss‚3.281151328768049 /loss/FBt4/detection_loss‚1.3301123347935693 /loss/FRt2/detection_loss‚3.3094631948254327 /loss/FBt1/detection_loss‚2.4763091146945952 trainer/global_step‚1649 $ _timestamp‚1.788531060552585e+09 %loss/total_loss‚51.22304603130313†Uîn‘š‚(:† þ¢ëÔ¿éÛ™ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3172 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ßé=r›‚(:ç þ¢ëÔ®…û( disk.nvme0c0n1.out‚39565.88232421875 disk.dm-0.in‚ 4709.39453125# disk.dm-0.out‚39565.88232421875 network.sent‚ 2.913364e+06) disk./.usagePercent‚79.00743953798381 disk.sda.in‚0 disk.sda.out‚65.71875! disk.nvme0n1.in‚ 4709.39453125 disk.sda1.in‚0 disk.drbd1260.in‚0$ memory_percent‚90.33143520355225 network.recv‚709290 disk.sda2.out‚0 disk.dm-1.out‚65.71875 disk.drbd1260.out‚ 0.28515625# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 cpu‚25.28994121019998% disk./.usageGB‚1177.2679443359375 disk.dm-1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1584.09765625 proc.cpu.threads‚17 disk.sda2.in‚0& disk.nvme0n1.out‚39565.88232421875 disk.sda3.in‚0 disk.sda3.out‚65.71875# disk.nvme0c0n1.in‚ 4709.39453125c" ‘œ‚(:† £ëԼ㗠gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.176' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8352 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>Ýåìy‚(:î £ëÔÜÉïš) disk./.usagePercent‚79.00743953798381# disk.nvme0c0n1.in‚ 4709.39453125( disk.nvme0c0n1.out‚39691.73388671875 disk.dm-0.in‚ 4709.39453125 disk.dm-1.out‚ 66.16015625 disk.sda1.in‚0 disk.sda2.in‚0% disk./.usageGB‚1177.2679443359375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.28515625 disk.sda2.out‚0& disk.nvme0n1.out‚39691.73388671875& proc.memory.availableMB‚ 1584.09375! disk.nvme0n1.in‚ 4709.39453125 disk.sda3.in‚0 disk.sda3.out‚ 66.16015625 cpu‚25.288548897464285 proc.cpu.threads‚17 network.sent‚ 3.021768e+06# disk.dm-0.out‚39691.73388671875 disk.sda.in‚0 disk.sda.out‚ 66.16015625$ memory_percent‚90.33145904541016# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652 network.recv‚7159290»xœž‚(:‘ œ£ëÔ’ã¿ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹©ŸçvŸ‚(:ë œ£ëԆƩ disk.dm-0.in‚ 4763.44140625# memory_percent‚90.3329849243164 proc.cpu.threads‚17 network.recv‚719203# disk.nvme0c0n1.in‚ 4763.44140625 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 1583.84375# proc.memory.rssMB‚ 3020.05859375* proc.memory.percent‚18.432974815368652) disk./.usagePercent‚79.00743953798381! disk.nvme0n1.in‚ 4763.44140625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2890625 disk.sda.out‚ 66.58203125 network.sent‚ 3.026699e+06& disk.nvme0n1.out‚39827.81494140625 disk.sda3.in‚0# disk.dm-0.out‚39827.81494140625 disk.dm-1.out‚ 66.58203125 disk.sda1.out‚0 disk.sda.in‚0 cpu‚25.28709999253099% disk./.usageGB‚1177.2679443359375 disk.sda3.out‚ 66.58203125( disk.nvme0c0n1.out‚39827.81494140625…]N) ‚(:… «£ëÔ…óµ— gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó ÑÞU¡‚(:Ê «£ëÔÄØÀœ disk.sda3.in‚0) disk./.usagePercent‚79.00738756845436 disk.dm-0.out‚40232.14453125 disk.sda2.out‚0! disk.nvme0n1.in‚ 4846.66796875# proc.memory.rssMB‚ 3020.05859375 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.2890625 network.sent‚ 3.032918e+06# disk.nvme0c0n1.in‚ 4846.66796875% disk.nvme0c0n1.out‚40232.14453125 disk.sda2.in‚0$ memory_percent‚90.33608436584473( proc.memory.availableMB‚ 1583.3359375 cpu‚25.28597751453342% disk./.usageGB‚1177.2671699523926 disk.dm-0.in‚ 4846.66796875 disk.drbd1260.in‚0 disk.sda.out‚67 disk.sda3.out‚67* proc.memory.percent‚18.432974815368652 network.recv‚723719 disk.dm-1.in‚0 disk.dm-1.out‚67 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0# disk.nvme0n1.out‚40232.14453125jŒ2¢- _wandbruntime‚2103  _runtime‚2103oŽ‹œÆ£À & lr-AdamW/pg1‚0.0008125320178806817 & lr-AdamW/pg2‚0.0008125320178806817 trainer/global_step‚1699 % _timestamp‚1.7885311192588093e+09 _runtime‚ 2103.41236851 _step‚135ÄÏ@”ˤŠtrainer/global_step‚1699 % _timestamp‚1.7885311192588093e+09 _runtime‚ 2103.41236851 _step‚135 & lr-AdamW/pg1‚0.0008125320178806817 & lr-AdamW/pg2‚0.0008125320178806817‡ã«z–2¥- _wandbruntime‚2103  _runtime‚2103fuV¢‡¦ÂO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72792639019520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3368131834368 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68784043480161 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72066266003968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12172802 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78090387 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68698152172544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3428037438976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68698152172544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3368365501952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65425454489456 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3368131834368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_byk}TAtes.small_pool.allocated‚ 3428332253696 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46041332 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32049057 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32049057 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72066517674496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68698134169600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19108084 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19108086 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76220995213312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19108086 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76220995213312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72066266003968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12172802 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78090387 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68698152172544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3428037438976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68698152172544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3368365501952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65425454489456 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3368131834368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3428332253696 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46041332 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32049057 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32049057 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72066517674496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68698134169600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19108084ˆQl—2¨- _wandbruntime‚2104  _runtime‚2104\n¼^Ò&©ÌM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72107767750656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31298844 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76264356097024 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72107516066816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65463001886240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78161724 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31299547 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3360547479691 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72107767750656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78135354 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32067330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3430321617408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65462984846880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12180659 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚1699 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32067332 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46068024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370095860224 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78161724 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46094392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76264668547072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72834322989568 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68823532326571 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3360776460975 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72834346929664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68737438209536 _step‚137 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚2104.493804884 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46094392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19118888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68737438209536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370329541120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370329541120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78135354 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46068024 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32067330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ðƒö%˪ÑM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46068024 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32067330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72107767750656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31298844 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76264356097024 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72107516066816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65463001886240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78161724 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31299547 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3360547479691 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72107767750656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78135354 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32067330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3430321617408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65462984846880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12180659 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚1699 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32067332 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46068024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370095860224 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78161724 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46094392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76264668547072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72834322989568 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68823532326571 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3360776460975 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72834346929664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68737438209536 _step‚137 TEDevic.ªM eStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚2104.493804884 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46094392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19118888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68737438209536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370329541120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370329541120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78135354 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284‰%Ý2«- _wandbruntime‚2104  _runtime‚2104á¶m]¬× /loss/FRt4/assignment_loss‚4.067757396862425 1loss/SRqqt4/detection_loss‚2.7450130581855774 .loss/FBt1/detection_loss‚2.632235680307661 epoch‚1 _runtime‚2104.495490111 2loss/FBt3/assignment_loss‚0.024603977600598625 % _timestamp‚1.7885311203421292e+09 0loss/FRt1/assignment_loss‚3.6327029498847754 0loss/FRt3/assignment_loss‚1.5989924157169504 1loss/FBt4/assignment_loss‚0.14944654882920788 /loss/FRt2/detection_loss‚2.9201011907669807 1loss/SRqqt2/detection_loss‚2.9001286293451605 0loss/FBt1/assignment_loss‚0.4595324253064713 /loss/FRt3/detection_loss‚1.6257116013849284 1loss/SRqqt3/assignment_loss‚1.379489496832007 /loss/FRt1/detection_loss‚2.3431535347087964 1loss/SRqqt3/detection_loss‚1.1194407019507269 /loss/FBt2/detection_loss‚2.3809978926883026 .loss/FBt3/detection_loss‚1.792533048335463 /loss/FBt4/detection_loss‚1.3774002047048675 trainer/global_step‚1699 _step‚138 1loss/FBt2/assignment_loss‚0.36706216029819294 %loss/total_loss‚48.21441297860766 0loss/FRt2/assignment_loss‚3.9982143006017132 1loss/SRqqt1/detection_loss‚1.9798168442970099 2loss/SRqqt1/assignment_loss‚1.4145489003249379 1loss/SRqqt2/assignment_loss‚1.993142905430152 1loss/SRqqt4/assignment_loss‚2.357721868563782 /loss/FRt4/detection_loss‚2.9546652456809737^³Yb­Ü 2loss/SRqqt1/assignment_loss‚1.4145489003249379 1loss/SRqqt2/assignment_loss‚1.993142905430152 1loss/SRqqt4/assignment_loss‚2.357721868563782 /loss/FRt4/detection_loss‚2.9546652456809737 /loss/FRt4/assignment_loss‚4.067757396862425 1loss/SRqqt4/detection_loss‚2.7450130581855774 .loss/FBt1/detection_loss‚2.632235680307661 epoch‚1 _runtime‚2104.495490111 2loss/FBt3/assignment_loss‚0.024603977600598625 % _timestamp‚1.7885311203421292e+09 0loss/FRt1/assignment_loss‚3.6327029498847754 0loss/FRt3/assignment_loss‚1.5989924157169504 1loss/FBt4/assignment_loss‚0.14944654882920788 /loss/FRt2/detection_loss‚2.9201011907669807 1loss/SRqqt2/detection_loss‚2.9001286293451605 0loss/FBt1/assignment_loss‚0.4595324253064713 /loss/FRt3/detection_loss‚1.6257116013849284 1loss/SRqqt3/assignment_loss‚1.379489496832007 /loss/FRt1/detection_loss‚2.3431535347087964 1loss/SRqqt3/detection_loss‚1.1194407019507269 /loss/FBt2/detection_loss‚2.3809978926883026 .loss/FBt3/detection_loss‚1.792533048335463 /loss/FBt4/detection_loss‚1.3774002047048675 trainer/global_step‚1699 _step‚138 1loss/FBt2/assignment_loss‚0.36706216029819294 %loss/total_loss‚48.21441297860766 0loss/FRt2/assignment_loss‚3.9982143006017132 1loss/SRqqt1/detection_loss‚1.9798168442970099Šrå‘®‚(:† º£ëԻŘ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0026 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Ti¯‚(:Þ º£ëÔ´œÁ› proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 3.037759e+06 network.recv‚727070) disk./.usagePercent‚79.00745182634546 disk.sda1.in‚0" disk.nvme0c0n1.out‚ 40373.96875 disk.dm-0.in‚ 4846.66796875! disk.nvme0n1.in‚ 4846.66796875 disk.dm-0.out‚ 40373.96875 disk.dm-1.out‚ 67.39453125$ memory_percent‚90.33341407775879* proc.memory.percent‚18.432974815368652 cpu‚25.284703945837542% disk./.usageGB‚1177.2681274414062 disk.sda1.out‚0 disk.drbd1260.out‚ 0.29296875 disk.sda.in‚0# disk.nvme0c0n1.in‚ 4846.66796875( proc.memory.availableMB‚ 1583.7734375# proc.memory.rssMB‚ 3020.05859375 disk.drbd1260.in‚0 disk.sda.out‚ 67.39453125 disk.nvme0n1.out‚ 40373.96875 disk.sda3.in‚0 disk.sda3.out‚ 67.39453125 disk.dm-1.in‚0yHÑ‘°‚(:† É£ëÔÄ·ì— gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2244 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÿÜf±‚(:Û É£ëÔ¦éáš cpu‚25.283673656138554 proc.cpu.threads‚17# disk.nvme0n1.out‚40894.95703125 disk.dm-0.in‚ 4846.66796875 disk.dm-1.in‚0 disk.dm-1.out‚67.8125 disk.sda1.out‚0 disk.sda.out‚67.8125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚40894.95703125 network.sent‚ 3.146163e+06 network.recv‚733607% disk./.usageGB‚1177.2680969238281 disk.sda3.in‚0$ memory_percent‚90.33350944519043( proc.memory.availableMB‚ 1583.7578125* proc.memory.percent‚18.432974815368652# disk.nvme0c0n1.in‚ 4846.66796875 disk.sda1.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 3020.05859375) disk./.usagePercent‚79.00744977828519! disk.nvme0n1.in‚ 4846.66796875 disk.sda3.out‚67.8125% disk.nvme0c0n1.out‚40894.95703125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.29296875˜*#`‘²‚(:† Ø£ëÔŒþ«• gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.601' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3202 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄfŒKd³‚(:٠أëÔ¬ð—$ disk./.usageGB‚1177.268856048584 disk.sda1.out‚0 disk.sda.out‚68.4375$ memory_percent‚90.33656120300293( proc.memory.availableMB‚ 1583.2578125* proc.memory.percent‚18.432998657226562 cpu‚25.282365377869812 disk.dm-1.out‚68.4375 disk.sda1.in‚0 network.recv‚737116# disk.nvme0c0n1.in‚ 4846.66796875 disk.dm-0.in‚ 4846.66796875 disk.sda2.out‚0! disk.nvme0n1.in‚ 4846.66796875% disk.nvme0n1.out‚41302.4560546875' disk.nvme0c0n1.out‚41302.4560546875 disk.sda2.in‚0 proc.memory.rssMB‚ 3020.0625 proc.cpu.threads‚17 network.sent‚ 3.151042e+06 disk.sda3.in‚0 disk.sda3.out‚68.4375" disk.dm-0.out‚41302.4560546875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.296875 disk.sda.in‚0( disk./.usagePercent‚79.0075007237845Íõ‰œ´‚(:‘ ç£ëÔÓæÙ— gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.205' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.041000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©¦g|hµ‚(:Ý ç£ëÔëÀ‘œ disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚1177.206199645996 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.296875 cpu‚25.281131798030174 disk.sda.out‚ 68.93359375 disk.sda2.out‚0 disk.nvme0n1.in‚ 4970.7890625% disk.nvme0n1.out‚41766.6396484375 disk.sda3.out‚ 68.93359375 disk.dm-0.in‚ 4970.7890625" disk.dm-0.out‚41766.6396484375 disk.dm-1.in‚0 network.recv‚741555( disk./.usagePercent‚79.0032958000346 disk.sda3.in‚0' disk.nvme0c0n1.out‚41766.6396484375$ memory_percent‚90.33966064453125# proc.memory.availableMB‚1582.75 proc.memory.rssMB‚ 3020.0625* proc.memory.percent‚18.432998657226562 network.sent‚ 3.157138e+06 disk.sda.in‚0" disk.nvme0c0n1.in‚ 4970.7890625 disk.dm-1.out‚ 68.93359375z©[ö2¶- _wandbruntime‚2162  _runtime‚2162XtÈáÇ·Á & lr-AdamW/pg2‚0.0008116588063287872 trainer/global_step‚1749 % _timestamp‚1.7885311783803315e+09 _runtime‚2162.533815065 _step‚139 & lr-AdamW/pg1‚0.0008116588063287872б:Ì¸Æ % _timestamp‚1.7885311783803315e+09 _runtime‚2162.533815065 _step‚139 & lr-AdamW/pg1‚0.0008116588063287872 & lr-AdamW/pg2‚0.0008116588063287872 trainer/global_step‚1749‹ŸÖM2¹- _wandbruntime‚2162  _runtime‚2162K¯ìÇ'ºÁO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74128783560192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚140 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80368433 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12519946 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74129035230720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19648184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3466347540992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78405030050304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚70662454022144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74876861462016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12520628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74128783560192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32168130 _runtime‚2162.535177494´aÌ'»ÆO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3466581208576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74128783560192 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚140 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80368433 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12519946 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74129035230720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19648184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3466347540992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78405030050304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚70662454022144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74876861462016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12520628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74128783560192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32168130 _runtime‚2162.535177494 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 IDeviceStatsMonitor.on_train_batc}5Œj&h_end/allocation.all.allocated‚80413426 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3468311566848 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 % _timestamp‚1.7885311794763477e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32185468 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67339519757440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458485520199 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19658986 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78448702996992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47405976 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3458714501483 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2163.631253817 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80387056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3468545247744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32981082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚141 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74170285306880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3468311566848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80387056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32981082 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47432344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70701740059136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47432344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74170285306880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3468545247744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19658988 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3530157564928 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70701722056192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74918545432064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3529873249280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70701722056192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70701740059136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47405976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70797988238279 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80413426 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12527116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12526482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70798234258923 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840óù?×&¾ÑM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67339519757440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458485520199 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19658986 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78448702996992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47405976 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3458714501483 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2163.631253817 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80387056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3468545247744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32981082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚141 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74170285306880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3468311566848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80387056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32981082 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47432344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70701740059136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47432344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74170285306880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3468545247744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19658988 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3530157564928 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70701722056192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74918545432064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3529873249280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70701722056192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70701740059136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47405976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70797988238279 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80413426 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12527116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12526482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70798234258923 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74170033623040 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80413426 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3468311566848 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 % _timestamp‚1.7885311794763477e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32185468 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634§à„Ê2¿- _wandbruntime‚2163  _runtime‚2163ŠÁ[ÀÕ /loss/FRt3/assignment_loss‚1.866469644529875 0loss/FRt4/assignment_loss‚3.8785433123509088 1loss/SRqqt1/assignment_loss‚1.353740970293681 .loss/FRt3/detection_loss‚1.516723372302083 0loss/SRqqt1/detection_loss‚2.284595975370118 0loss/SRqqt4/detection_loss‚2.896690411567688 % _timestamp‚1.7885311794789603e+09 /loss/FRt1/assignment_loss‚4.644942067563534 0loss/FBt4/assignment_loss‚0.3222617502904987 .loss/FRt1/detection_loss‚2.547815952450037 /loss/FRt2/detection_loss‚3.0334757516781488 2loss/SRqqt3/assignment_loss‚1.2953880460844154 0loss/FBt1/assignment_loss‚0.5378561005846643 4loss/FBt3/assignment_loss‚1.0482659582046318e-05 epoch‚1 _step‚142 /loss/FRt2/assignment_loss‚4.348399817943573 1loss/SRqqt3/detection_loss‚0.9199426613748074 /loss/FBt1/detection_loss‚2.8165811483676615 &loss/total_loss‚48.019944492387914 trainer/global_step‚1749 _runtime‚2163.632350485 2loss/SRqqt2/assignment_loss‚1.8114112329979737 1loss/SRqqt2/detection_loss‚2.5313412845134735 2loss/SRqqt4/assignment_loss‚1.9331470885872841 0loss/FBt2/assignment_loss‚0.2664989034883392 /loss/FBt3/detection_loss‚0.7951053720381525 /loss/FBt4/detection_loss‚1.1385509667672076 .loss/FRt4/detection_loss‚2.657615201340781 /loss/FBt2/detection_loss‚2.6228369772434235»%z`ÁÚ /loss/FRt1/assignment_loss‚4.644942067563534 0loss/FBt4/assignment_loss‚0.3222617502904987 .loss/FRt1/detection_loss‚2.547815952450037 /loss/FRt2/detection_loss‚3.0334757516781488 2loss/SRqqt3/assignment_loss‚1.2953880460844154 0loss/FBt1/assignment_loss‚0.5378561005846643 4loss/FBt3/assignment_loss‚1.0482659582046318e-05 epoch‚1 _step‚142 /loss/FRt2/assignment_loss‚4.348399817943573 1loss/SRqqt3/detection_loss‚0.9199426613748074 /loss/FBt1/detection_loss‚2.8165811483676615 &loss/total_loss‚48.019944492387914 trainer/global_step‚1749 _runtime‚2163.632350485 2loss/SRqqt2/assignment_loss‚1.8114112329979737 1loss/SRqqt2/detection_loss‚2.5313412845134735 2loss/SRqqt4/assignment_loss‚1.9331470885872841 0loss/FBt2/assignment_loss‚0.2664989034883392 /loss/FBt3/detection_loss‚0.7951053720381525 /loss/FBt4/detection_loss‚1.1385509667672076 .loss/FRt4/detection_loss‚2.657615201340781 /loss/FBt2/detection_loss‚2.6228369772434235 /loss/FRt3/assignment_loss‚1.866469644529875 0loss/FRt4/assignment_loss‚3.8785433123509088 1loss/SRqqt1/assignment_loss‚1.353740970293681 .loss/FRt3/detection_loss‚1.516723372302083 0loss/SRqqt1/detection_loss‚2.284595975370118 0loss/SRqqt4/detection_loss‚2.896690411567688 % _timestamp‚1.7885311794789603e+09Ž­Hܑ‚(:† ö£ëÔù¾Ÿ— gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.483' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2966 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f‡G—hÂ(:Ý ö£ëÔ«Ï™ disk.nvme0n1.in‚ 4970.7890625# disk.nvme0n1.out‚41914.60546875 disk.sda3.in‚0 disk.dm-0.in‚ 4970.7890625 disk.dm-0.out‚41914.60546875 disk.sda1.in‚0$ memory_percent‚90.33527374267578 network.sent‚ 3.161988e+06 disk.sda.in‚0 disk.dm-1.out‚ 69.37109375 disk.drbd1260.out‚ 0.30078125 cpu‚25.27996248045796 proc.cpu.threads‚17 network.recv‚745005% disk./.usageGB‚1176.7089309692383 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 69.37109375" disk.nvme0c0n1.in‚ 4970.7890625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 69.37109375% disk.nvme0c0n1.out‚41914.60546875 disk.sda1.out‚0& proc.memory.availableMB‚ 1583.46875 proc.memory.rssMB‚ 3020.0625* proc.memory.percent‚18.432998657226562) disk./.usagePercent‚78.96992368190118¢•~›Ä‚(: …¤ëÔÐÐã— gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚96.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.264799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a‹77SÅ‚(:È …¤ëÔ”êš disk.sda.out‚ 70.015625 disk.nvme0n1.in‚ 4970.7890625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0* proc.memory.percent‚18.432998657226562 network.sent‚ 3.270744e+06) disk./.usagePercent‚78.96983382325666 disk.sda3.out‚ 70.015625 disk.dm-0.out‚ 41990.125$ memory_percent‚90.33527374267578& proc.memory.availableMB‚ 1583.46875 proc.memory.rssMB‚ 3020.0625 network.recv‚751666$ disk./.usageGB‚1176.707592010498" disk.nvme0c0n1.in‚ 4970.7890625 disk.nvme0c0n1.out‚ 41990.125 disk.dm-0.in‚ 4970.7890625 disk.dm-1.out‚ 70.015625 disk.sda1.in‚0 disk.drbd1260.out‚ 0.30078125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.out‚ 41990.125 cpu‚25.278628628988663 proc.cpu.threads‚17#wd@‘Æ‚(:† ”¤ëÔÜ°ü– gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚141.574' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3148 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~›@qÇ‚(:æ ”¤ëÔ̲ӗ( disk.nvme0c0n1.out‚42326.04931640625 disk.dm-1.out‚ 70.41796875 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚90.33222198486328* proc.memory.percent‚18.432998657226562# disk.dm-0.out‚42326.04931640625 disk.sda.out‚ 70.41796875& disk.nvme0n1.out‚42326.04931640625 disk.sda3.out‚ 70.41796875& proc.memory.availableMB‚ 1583.96875 proc.cpu.threads‚17 network.sent‚ 3.277534e+06 disk.dm-0.in‚ 4970.7890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.277599817233725 network.recv‚755201) disk./.usagePercent‚78.96983510329433" disk.nvme0c0n1.in‚ 4970.7890625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.3046875 disk.sda.in‚0 disk.nvme0n1.in‚ 4970.7890625 proc.memory.rssMB‚ 3020.0625% disk./.usageGB‚1176.7076110839844ä©mÈ‚(:… £¤ëÔÔ”¨— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.645' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.929 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðú;^oÉ‚(:ä £¤ëÔµÚ€› disk.sda3.out‚ 70.8671875 disk.dm-0.in‚ 5078.0546875 disk.sda.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚42609.25634765625$ memory_percent‚90.33694267272949( proc.memory.availableMB‚ 1583.1953125 proc.memory.rssMB‚ 3020.0625 disk.sda1.in‚0) disk./.usagePercent‚78.96983459127927 disk.sda3.in‚0 disk.dm-1.in‚0 cpu‚25.27645218259642( disk.nvme0c0n1.out‚42609.25634765625 disk.drbd1260.in‚0 disk.sda.out‚ 70.8671875 disk.sda2.out‚0 disk.nvme0n1.in‚ 5078.0546875* proc.memory.percent‚18.432998657226562 proc.cpu.threads‚17 disk.dm-1.out‚ 70.8671875 disk.sda1.out‚0 network.sent‚ 3.281783e+06 network.recv‚759817% disk./.usageGB‚1176.7076034545898" disk.nvme0c0n1.in‚ 5078.0546875# disk.dm-0.out‚42609.25634765625 disk.drbd1260.out‚ 0.3046875Ì¥™2Ê- _wandbruntime‚2221  _runtime‚2221ÒÊÇËÁ trainer/global_step‚1799 % _timestamp‚1.7885312372334769e+09 _runtime‚2221.387308285 _step‚143 & lr-AdamW/pg1‚0.0008107015365851014 & lr-AdamW/pg2‚0.0008107015365851014×^ ¯ÌÌÆ _runtime‚2221.387308285 _step‚143 & lr-AdamW/pg1‚0.0008107015365851014 & lr-AdamW/pg2‚0.0008107015365851014 trainer/global_step‚1799 % _timestamp‚1.7885312372334769e+097dó2Í- _wandbruntime‚2221  _runtime‚2221~ß%ÞÎÁO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69178490231856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76191552786944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12868151 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3627999669760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3627709049344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20188284 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76961036024320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72733201324013 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 _runtime‚2221.389102808 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33876555 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80588745073664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76961059964416 $ _timestamp‚1.788531237235239e+09 _step‚144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69178507271216 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76191301116416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3564796915200 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72732955303577 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48717236 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12868799 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80589059634176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3564563247616 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76191552786944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76191301116416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33876557 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72626737868800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33876557 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82620135 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82593791 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3564563247616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72626755871744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3564796915200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3564563247616 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76191552786944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76191301116416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33876557 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72626737868800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33876557 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82620135 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82593791 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3564563247616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72626755871744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3564796915200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33074143 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48743928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76232802863104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80632429856768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3556652541991 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72666023905792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20199088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82665128 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚1799 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3566760954368 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82638758 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77002743934464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33894832 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48743928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82638758 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3556423560707 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2222.634540025 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69216037628640 _step‚145 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48770296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚330748163¶…í%ÒÐM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33074143 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48743928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76232802863104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80632429856768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3556652541991 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72666023905792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20199088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82665128 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚1799 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3566760954368 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82638758 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77002743934464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33894832 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48743928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.currentÌç u± ‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82638758 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3556423560707 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2222.634540025 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69216037628640 _step‚145 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48770296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33074816‘Ì'92Ó- _wandbruntime‚2222  _runtime‚2222»qÙ.YÔÓ /loss/FRt1/assignment_loss‚4.489629580128577 1loss/FBt1/assignment_loss‚0.40093192547611317 /loss/FRt2/detection_loss‚3.0462701869010926 /loss/FBt1/detection_loss‚2.5640100283282146 trainer/global_step‚1799 $ _timestamp‚1.788531238482499e+09 /loss/FRt4/assignment_loss‚4.276347881363284 3loss/FBt3/assignment_loss‚0.0035972096025886913 /loss/FRt3/detection_loss‚1.5235489509999751 1loss/SRqqt3/assignment_loss‚1.207273996509282 /loss/FRt4/detection_loss‚2.8890584841851266 .loss/FBt4/detection_loss‚2.019701548884141 2loss/SRqqt1/assignment_loss‚1.8753541368418014 .loss/FBt2/detection_loss‚2.607211448252201 /loss/FBt3/detection_loss‚0.7519462818900744 /loss/FRt1/detection_loss‚2.7496605967321703 1loss/SRqqt3/detection_loss‚0.9641221822363085 _step‚146 1loss/SRqqt1/detection_loss‚2.0311407492710996 epoch‚1 /loss/FRt2/assignment_loss‚3.663463308811188 0loss/FRt3/assignment_loss‚1.5199899514516195 1loss/SRqqt2/assignment_loss‚2.209017397960027 0loss/FBt2/assignment_loss‚0.1892877728808955 1loss/FBt4/assignment_loss‚0.21708532956349583 0loss/SRqqt4/assignment_loss‚2.00488114496693 %loss/total_loss‚48.18028595070386 _runtime‚2222.636000709 1loss/SRqqt2/detection_loss‚2.6025283336639404 0loss/SRqqt4/detection_loss‚2.374227523803711tÄã ^ÕØ /loss/FRt1/detection_loss‚2.7496605967321703 1loss/SRqqt3/detection_loss‚0.9641221822363085 _step‚146 1loss/SRqqt1/detection_loss‚2.0311407492710996 epoch‚1 /loss/FRt2/assignment_loss‚3.663463308811188 0loss/FRt3/assignment_loss‚1.5199899514516195 1loss/SRqqt2/assignment_loss‚2.209017397960027 0loss/FBt2/assignment_loss‚0.1892877728808955 1loss/FBt4/assignment_loss‚0.21708532956349583 0loss/SRqqt4/assignment_loss‚2.00488114496693 %loss/total_loss‚48.18028595070386 _runtime‚2222.636000709 1loss/SRqqt2/detection_loss‚2.6025283336639404 0loss/SRqqt4/detection_loss‚2.374227523803711 /loss/FRt1/assignment_loss‚4.489629580128577 1loss/FBt1/assignment_loss‚0.40093192547611317 /loss/FRt2/detection_loss‚3.0462701869010926 /loss/FBt1/detection_loss‚2.5640100283282146 trainer/global_step‚1799 $ _timestamp‚1.788531238482499e+09 /loss/FRt4/assignment_loss‚4.276347881363284 3loss/FBt3/assignment_loss‚0.0035972096025886913 /loss/FRt3/detection_loss‚1.5235489509999751 1loss/SRqqt3/assignment_loss‚1.207273996509282 /loss/FRt4/detection_loss‚2.8890584841851266 .loss/FBt4/detection_loss‚2.019701548884141 2loss/SRqqt1/assignment_loss‚1.8753541368418014 .loss/FBt2/detection_loss‚2.607211448252201 /loss/FBt3/detection_loss‚0.7519462818900744’Bߛւ(: ²¤ëÔĞؖ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.797' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.159399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•*krׂ(:ç ²¤ëÔŠñ—˜ disk.sda2.out‚0% disk.nvme0n1.out‚43071.4599609375" disk.dm-0.out‚43071.4599609375 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚90.33586978912354 disk.sda3.out‚ 71.37890625" disk.nvme0c0n1.in‚ 5078.0546875 disk.dm-1.out‚ 71.37890625 disk.drbd1260.out‚ 0.30859375 proc.memory.rssMB‚ 3020.0625* proc.memory.percent‚18.432998657226562 cpu‚25.275242406251607 proc.cpu.threads‚17 network.sent‚ 3.286605e+06% disk./.usageGB‚1176.6301460266113 disk.sda.out‚ 71.37890625 disk.sda3.in‚0' disk.nvme0c0n1.out‚43071.4599609375 disk.dm-1.in‚0 disk.sda1.in‚0 network.recv‚763456) disk./.usagePercent‚78.96463635829652 disk.nvme0n1.in‚ 5078.0546875 disk.dm-0.in‚ 5078.0546875) proc.memory.availableMB‚ 1583.37109375 disk.sda.in‚0 disk.sda2.in‚0÷.#CœØ‚(:‘ Á¤ëÔóÏç• gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.520599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆl¸gÙ‚(:Ü Á¤ëÔÙ§¤˜' disk.nvme0c0n1.out‚43330.5068359375" disk.dm-0.out‚43330.5068359375 disk.sda1.in‚0 disk.nvme0n1.in‚ 5078.0546875 disk.sda3.in‚0 disk.sda3.out‚72.09375 disk.dm-0.in‚ 5078.0546875 disk.dm-1.out‚72.09375 disk.drbd1260.out‚ 0.30859375 disk.sda2.out‚0% disk.nvme0n1.out‚43330.5068359375# memory_percent‚90.3343915939331) proc.memory.availableMB‚ 1583.61328125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚72.09375 disk.sda2.in‚0 network.sent‚ 3.395049e+06) disk./.usagePercent‚78.96306139994587" disk.nvme0c0n1.in‚ 5078.0546875 disk.drbd1260.in‚0 proc.memory.rssMB‚ 3020.0625* proc.memory.percent‚18.432998657226562 cpu‚25.27419843816992 proc.cpu.threads‚17 network.recv‚770185% disk./.usageGB‚1176.6066780090332¨\“yÚ‚(:… ФëÔ¯ð‚• gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.185' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.637 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~C­àdÛ‚(:Ù ФëÔ÷¸Ìš network.recv‚773540) disk./.usagePercent‚78.96463635829652 disk.dm-1.out‚ 72.5546875 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 5078.0546875 proc.memory.rssMB‚ 3020.0625 network.sent‚ 3.399924e+06 disk.sda1.in‚0 disk.drbd1260.out‚0.3125 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0% disk.nvme0c0n1.out‚43671.64453125 disk.dm-0.in‚ 5078.0546875% disk./.usageGB‚1176.6301460266113 disk.dm-1.in‚0 disk.sda.out‚ 72.5546875# disk.nvme0n1.out‚43671.64453125 disk.dm-0.out‚43671.64453125$ memory_percent‚90.33446311950684 cpu‚25.273010241860415 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 5078.0546875 disk.sda3.out‚ 72.5546875( proc.memory.availableMB‚ 1583.6015625* proc.memory.percent‚18.432998657226562kŽz՛܂(: ߤëÔˆñ ˜ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.146' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.829199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z±hßfÝ‚(:Û ߤëÔ‡”ø™% disk./.usageGB‚1176.6302299499512 disk.dm-0.in‚ 5202.9453125 disk.dm-1.out‚ 73.12890625 disk.sda1.in‚0 disk.drbd1260.in‚0# memory_percent‚90.3374195098877% disk.nvme0c0n1.out‚43762.47265625 disk.sda.in‚0 disk.sda.out‚ 73.12890625* proc.memory.percent‚18.432998657226562 proc.cpu.threads‚17 network.sent‚ 3.407917e+06" disk.nvme0c0n1.in‚ 5202.9453125 disk.dm-0.out‚43762.47265625 disk.sda1.out‚0 disk.drbd1260.out‚0.3125# disk.nvme0n1.out‚43762.47265625 disk.sda3.in‚0( proc.memory.availableMB‚ 1583.1171875 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 5202.9453125 disk.sda3.out‚ 73.12890625 proc.memory.rssMB‚ 3020.0625 cpu‚25.271904960412197 network.recv‚778311) disk./.usagePercent‚78.96464199046227o$là2Þ- _wandbruntime‚2280  _runtime‚2280ïx”Åß¿ % lr-AdamW/pg1‚0.000809660408143204 % lr-AdamW/pg2‚0.000809660408143204 trainer/global_step‚1849 % _timestamp‚1.7885312967738314e+09 _runtime‚2280.928534093 _step‚147°3aYÊàÄ % _timestamp‚1.7885312967738314e+09 _runtime‚2280.928534093 _step‚147 % lr-AdamW/pg1‚0.000809660408143204 % lr-AdamW/pg2‚0.000809660408143204 trainer/global_step‚1849“LêP)2á- _wandbruntime‚2280  _runtime‚2280çþþ!È'âÂO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50055188 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71055008103056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78254070343168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3652403112229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885312967770658e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78253818672640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34790307 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50055188 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3662778954240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74707657235721 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79045234526720 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78253818672640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84871837 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3663012621824 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78254070343168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84845493 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79045258466816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727876383232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71055025142416 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50081530 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74591057721344 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74707411215285 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2280.931470037M=bÍ'ãÇO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79045234526720 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78253818672640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84871837 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3663012621824 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78254070343168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84845493 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79045258466816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727876383232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71055025142416 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50081530 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74591057721344 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74707411215285 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2280.931470037 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50055188 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71055008103056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78254070343168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3652403112229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885312967770658e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78253818672640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34790307 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50055188 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3662778954240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74707657235721”x›ä2ä- _wandbruntime‚2281  _runtime‚2281®\ØåÌM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13220822 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33960010 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34808580 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78295320419328 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚8281681_À·#6752640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74747146082339 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33959320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84916830 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3664976660992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3664742980096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74630325755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74630325755392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84890460 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34808582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3664976660992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84890460 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84916830 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78295320419328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3654361601215 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50108248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74630343758336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 _runtime‚2281.950593129 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79086942436864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13220134 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3729874315776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20739186 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71092538460480 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50081880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34808582 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79086918496768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78295068735488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74746900061695 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3654590582499 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82816508496896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3729590000128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚149 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3664742980096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50081880y†sG×&æÑM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74630343758336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 _runtime‚2281.950593129 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79086942436864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13220134 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3729874315776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20739186 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71092538460480 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50081880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34808582 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79086918496768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78295068735488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74746900061695 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3654590582499 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82816508496896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3729590000128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚149 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3664742980096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50081880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13220822 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33960010 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34808580 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78295320419328 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82816816752640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74747146082339 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33959320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84916830 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3664976660992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3664742980096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74630325755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74630325755392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84890460 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34808582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3664976660992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84890460 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84916830 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78295320419328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3654361601215 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50108248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360•t®2ç- _wandbruntime‚2281  _runtime‚2281‡@3È[èÕ %loss/total_loss‚49.00560614371216 _runtime‚2281.951529645 2loss/SRqqt1/assignment_loss‚1.5636951916834765 1loss/SRqqt1/detection_loss‚1.8008006931396954 _step‚150 0loss/FRt1/assignment_loss‚4.3601988832155865 /loss/FRt2/assignment_loss‚4.910600662231445 1loss/FBt2/assignment_loss‚0.32088153912777767 2loss/FBt3/assignment_loss‚8.48169932169185e-05 1loss/SRqqt2/detection_loss‚3.0719106009131982 .loss/FRt4/detection_loss‚2.849509695091763 /loss/FBt3/detection_loss‚0.9048921116627753 /loss/FBt4/detection_loss‚1.2782495089057015 0loss/FRt3/assignment_loss‚1.9155211262888723 0loss/FBt1/assignment_loss‚0.3914434465347717 /loss/FRt3/detection_loss‚1.6693211354605564 trainer/global_step‚1849 1loss/SRqqt4/assignment_loss‚2.235823924894686 epoch‚1 0loss/SRqqt4/detection_loss‚2.361123207542631 /loss/FBt2/detection_loss‚2.3646459215217166 /loss/FRt4/assignment_loss‚4.219002385397215 0loss/FBt4/assignment_loss‚0.2851801171896668 % _timestamp‚1.7885312977981076e+09 2loss/SRqqt2/assignment_loss‚2.3718985789700557 2loss/SRqqt3/assignment_loss‚1.1271802672081523 /loss/FRt1/detection_loss‚2.3475859463214874 /loss/FRt2/detection_loss‚3.2321910858154297 1loss/SRqqt3/detection_loss‚1.1174122020602226 -loss/FBt1/detection_loss‚2.30645309554206ªp `éÚ /loss/FRt3/detection_loss‚1.6693211354605564 trainer/global_step‚1849 1loss/SRqqt4/assignment_loss‚2.235823924894686 epoch‚1 0loss/SRqqt4/detection_loss‚2.361123207542631 /loss/FBt2/detection_loss‚2.3646459215217166 /loss/FRt4/assignment_loss‚4.219002385397215 0loss/FBt4/assignment_loss‚0.2851801171896668 % _timestamp‚1.7885312977981076e+09 2loss/SRqqt2/assignment_loss‚2.3718985789700557 2loss/SRqqt3/assignment_loss‚1.1271802672081523 /loss/FRt1/detection_loss‚2.3475859463214874 /loss/FRt2/detection_loss‚3.2321910858154297 1loss/SRqqt3/detection_loss‚1.1174122020602226 -loss/FBt1/detection_loss‚2.30645309554206 %loss/total_loss‚49.00560614371216 _runtime‚2281.951529645 2loss/SRqqt1/assignment_loss‚1.5636951916834765 1loss/SRqqt1/detection_loss‚1.8008006931396954 _step‚150 0loss/FRt1/assignment_loss‚4.3601988832155865 /loss/FRt2/assignment_loss‚4.910600662231445 1loss/FBt2/assignment_loss‚0.32088153912777767 2loss/FBt3/assignment_loss‚8.48169932169185e-05 1loss/SRqqt2/detection_loss‚3.0719106009131982 .loss/FRt4/detection_loss‚2.849509695091763 /loss/FBt3/detection_loss‚0.9048921116627753 /loss/FBt4/detection_loss‚1.2782495089057015 0loss/FRt3/assignment_loss‚1.9155211262888723 0loss/FBt1/assignment_loss‚0.3914434465347717–›9>&‘ê‚(:† î¤ëÔØÆ„š gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.836' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3672 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ëoåaë‚(:Ö î¤ëÔÀȈ disk.sda.in‚0 disk.nvme0n1.in‚ 5202.9453125# disk.nvme0n1.out‚44143.00390625 disk.dm-0.in‚ 5202.9453125 disk.dm-1.in‚0 disk.dm-1.out‚ 73.890625 disk.sda2.out‚0 network.sent‚ 3.41094e+06 network.recv‚781836 disk.sda1.out‚0 disk.sda.out‚ 73.890625$ memory_percent‚90.33470153808594% proc.memory.availableMB‚ 1583.5625* proc.memory.percent‚18.432998657226562" disk.nvme0c0n1.in‚ 5202.9453125 proc.memory.rssMB‚ 3020.0625 cpu‚25.270835610061095 proc.cpu.threads‚17) disk./.usagePercent‚78.97367035216479% disk./.usageGB‚1176.7647590637207 disk.drbd1260.out‚ 0.31640625 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 73.890625% disk.nvme0c0n1.out‚44143.00390625 disk.dm-0.out‚44143.00390625 disk.sda1.in‚0 disk.drbd1260.in‚0F—b œì‚(:‘ ý¤ëԹ씖 gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.906' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.781200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê0+fí‚(:Û ý¤ëÔÃ% disk.nvme0c0n1.out‚44332.22265625 disk.dm-0.in‚ 5202.9453125 disk.sda.out‚ 74.25390625 disk.sda2.out‚0 disk.nvme0n1.in‚ 5202.9453125 cpu‚25.26976251635865% disk./.usageGB‚1176.7644157409668 disk.dm-1.in‚0 disk.dm-1.out‚ 74.25390625 disk.sda1.out‚0 disk.sda.in‚0# disk.nvme0n1.out‚44332.22265625 disk.sda3.in‚0$ memory_percent‚90.33470153808594 network.recv‚788868 disk.sda1.in‚0 disk.drbd1260.out‚ 0.31640625 disk.sda2.in‚0% proc.memory.availableMB‚ 1583.5625 proc.memory.rssMB‚ 3020.0625* proc.memory.percent‚18.432998657226562 proc.cpu.threads‚17 network.sent‚ 3.519538e+06" disk.nvme0c0n1.in‚ 5202.9453125 disk.dm-0.out‚44332.22265625 disk.drbd1260.in‚0 disk.sda3.out‚ 74.25390625( disk./.usagePercent‚78.9736473114867Ž]êœî‚(:‘ Œ¥ëÔœ«µ– gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.706599999999998 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p~çætï‚(:é Œ¥ëÔË™˜› network.recv‚792233 disk.nvme0n1.in‚ 5202.9453125 disk.drbd1260.out‚ 0.3203125$ memory_percent‚90.33472537994385) proc.memory.availableMB‚ 1583.55859375* proc.memory.percent‚18.432998657226562 disk.dm-0.in‚ 5202.9453125# disk.dm-0.out‚44619.37255859375) disk./.usagePercent‚78.97389589480245 disk.sda2.out‚0 disk.sda3.out‚ 74.71484375" disk.nvme0c0n1.in‚ 5202.9453125 disk.sda1.in‚0 proc.memory.rssMB‚ 3020.0625% disk./.usageGB‚1176.7681198120117 disk.sda1.out‚0 disk.drbd1260.in‚0 cpu‚25.268706965642252 proc.cpu.threads‚17 disk.sda2.in‚0& disk.nvme0n1.out‚44619.37255859375 disk.sda3.in‚0( disk.nvme0c0n1.out‚44619.37255859375 disk.dm-1.in‚0 disk.dm-1.out‚ 74.71484375 disk.sda.in‚0 disk.sda.out‚ 74.71484375 network.sent‚ 3.524403e+06Û%2ð- _wandbruntime‚2339  _runtime‚2339ÛÔë˜ÇñÁ & lr-AdamW/pg1‚0.0008085356379727022 & lr-AdamW/pg2‚0.0008085356379727022 trainer/global_step‚1899 % _timestamp‚1.7885313550785108e+09 _runtime‚2339.232234615 _step‚151lmÐËÌòÆ & lr-AdamW/pg1‚0.0008085356379727022 & lr-AdamW/pg2‚0.0008085356379727022 trainer/global_step‚1899 % _timestamp‚1.7885313550785108e+09 _runtime‚2339.232234615 _step‚151—¯g¼^2ó- _wandbruntime‚2339  _runtime‚2339 :²]ôÂO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3761228328448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21268486 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3827816354816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80316336228864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76555359570944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35704055 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76681867126993 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76555359570944 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34830934 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80316336228864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81129456969216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87097195 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51393140 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80316587899392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚7655534156*í!®8000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84956952472064 % _timestamp‚1.7885313550801888e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35704055 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80316587899392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72931543013616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51393140 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21268484 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81129433029120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3827519442944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3760994660864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76682113147429 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚1899 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76555359570944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35704055 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76681867126993 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76555359570944 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34830934 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80316336228864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81129456969216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87097195 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51393140 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80316587899392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76555341568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84956952472064 % _timestamp‚1.7885313550801888e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35704055 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80316587899392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72931543013616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51393140 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21268484 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81129433029120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3827519442944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3760994660864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76682113147429 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚1899 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3761228328448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21268486 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3827816354816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0˜okYm‘ö‚(:† ›¥ëÔÊûÀš gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚108.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%÷+ðm÷‚(:â ›¥ëÔ­òñ* proc.memory.percent‚18.432998657226562 network.recv‚797097! disk.nvme0c0n1.in‚ 5302.609375( disk.nvme0c0n1.out‚44960.15771484375# disk.dm-0.out‚44960.15771484375 disk.sda.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚44960.15771484375 cpu‚25.26758311047238 disk.sda3.out‚ 75.16796875 disk.sda1.in‚0 disk.sda.out‚ 75.16796875 proc.cpu.threads‚17 network.sent‚ 3.53051e+06) disk./.usagePercent‚78.97234039302441% disk./.usageGB‚1176.7449417114258 disk.dm-0.in‚ 5302.609375 disk.dm-1.in‚0 disk.dm-1.out‚ 75.16796875 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.3203125 disk.sda2.in‚0 disk.nvme0n1.in‚ 5302.609375# memory_percent‚90.3378963470459( proc.memory.availableMB‚ 1583.0390625 proc.memory.rssMB‚ 3020.0625 82ø- _wandbruntime‚2340  _runtime‚2340¯VÆ×Ò&ùÌM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚1899 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34849110 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35722330 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80357837975552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87168532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21279288 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76721601994047 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13569822 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80357586291712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829519934464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87168532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _step‚153 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51419832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21279286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722330 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81171140939264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3763192367616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34848407 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76594645607936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3762958686720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51446200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72969056331680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81171116999168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3752299641723 % _timestamp‚1.7885313561744068e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76594627604992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76594627604992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13569121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87142162 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76594645607936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51446200 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85000636933632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722332 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3763192367616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76594645607936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3762958686720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51446200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72969056331680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81171116999168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3752299641723 % _timestamp‚1.7885313561744068e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76594627604992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76594627604992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13569121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87142162 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76594645607936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51446200 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85000636933632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722332 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3763192367616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13569822 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80357586291712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829519934464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch¥eŸ… _end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87168532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _step‚153 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51419832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21279286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722330 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81171140939264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3763192367616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34848407 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82378853785088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78519661420544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36617807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13909493 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83213655471616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13910182 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82378853785088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1949 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82379105455616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78656323038701 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74808043845456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36617807 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89375241 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3848279193245 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83213631531520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚74808060884816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89348897 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52757434 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87140951523328 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78519643417600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82379105455616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3859444035072 _step‚156 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52731092 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35718768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87141270278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82378853785088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78519661420544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36617807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13909493 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83213655471616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13910182 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82378853785088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1949 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82379105455616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78656323038701 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74808043845456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36617807 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89375241 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3848279193245 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83213631531520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚74808060884816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89348897 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52757434 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87140951523328 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78519643417600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82379105455616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3859444035072 _step‚156 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52731092 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35718768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87141270278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0œ•–012Š- _wandbruntime‚2396  _runtime‚2396UÌ„/ ‹ËM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788531412774295e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 _runtime‚2396.929298528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89420234 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3850237682231 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35736235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚78696057905755 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52757784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3929308654080 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52784152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35735528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3929605552640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36636082 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82420355531776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78695811885111 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3850466663515 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36636080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3861408074240 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861174393344 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74845591242240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52757784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3929308654080 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52784152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35735528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3929605552640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36636082 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82420355531776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78695811885111 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3850466663515 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36636080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3861408074240 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861174393344 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74845591242240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 _runtime‚2396.929298528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89420234 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3850237682231 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35736235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚78696057905755 ODeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36608764 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54095386 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14260080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37531555 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3957659741696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36609453 cQDeviceStatsMoni‰Tžà® tor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4027372210688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚1999 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84441371341312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80483945267200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85297853974016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84441623011840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3946446214829 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89325226184704 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54095386 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76684561716656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80483963270144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37531555 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3957426074112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80483963270144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319?!„ðÍ'ÇO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91600599 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80483945267200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80631024970845 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runtime‚2454.560492002 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89324901138432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37531557 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36608764 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54095386 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14260080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37531555 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3957659741696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36609453 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4027372210688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚1999 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84441371341312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80483945267200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85297853974016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84441623011840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3946446214829 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89325226184704 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54095386 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76684561716656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80483963270144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37531555 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3957426074112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80483963270144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22359488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4029064016384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80670513817463 % _timestamp‚1.7885314716109278e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36627062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3959623780864 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84482873088000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85339537944064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3959390099968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22359486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36626391 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89368896761856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76722092074080 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚1999 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54122104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91645566 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89368578020352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4029358817792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76722109113440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3959623780864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14267574 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91671936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚2455.765840362 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3959390099968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91645566 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80523231304192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37549830 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80670267796819 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37549830 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80523249307136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14266905 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚161 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948175722739 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91671936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54095736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80523249307136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54122104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84482873088000 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54095736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54122104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84482873088000 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54095736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22359488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4029064016384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80670513817463 % _timestamp‚1.7885314716109278e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36627062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3959623780864 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84482873088000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85339537944064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3959390099968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22359486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36626391 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89368896761856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76722092074080 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚1999 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54122104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91645566 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89368578020352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4029358817792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76722109113440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3959623780864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14267574 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91671936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 GADeviceStatsWeýyMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚2455.765840362 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3959390099968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91645566 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582¡0DÈý2¡- _wandbruntime‚2455  _runtime‚2455\{ÌáX¢Ò 1loss/SRqqt1/detection_loss‚1.9138377867639065 2loss/SRqqt2/assignment_loss‚1.8324582850933075 2loss/SRqqt4/assignment_loss‚1.7864960005337542 0loss/FBt4/assignment_loss‚0.2132400894024651 epoch‚1 0loss/SRqqt3/assignment_loss‚1.23385250039639 1loss/FBt2/assignment_loss‚0.22755274739377285 /loss/FRt3/detection_loss‚1.5856477964317073 .loss/FRt4/detection_loss‚3.146388734762485 0loss/SRqqt4/detection_loss‚2.781117796897888 .loss/FBt3/detection_loss‚1.040469878539443 &loss/total_loss‚47.247168311613876 trainer/global_step‚1999 2loss/SRqqt1/assignment_loss‚1.6315442897379397 /loss/FRt2/assignment_loss‚4.105194061994553 0loss/FRt3/assignment_loss‚1.8391238918175568 3loss/FBt3/assignment_loss‚8.702166780949483e-07 .loss/FBt1/detection_loss‚2.382553704082966 _runtime‚2455.768203697 1loss/SRqqt2/detection_loss‚2.5471259260177614 .loss/FBt2/detection_loss‚2.268144767731428 0loss/FBt1/assignment_loss‚0.1836220922527616 /loss/FRt2/detection_loss‚2.9859500098973513 1loss/SRqqt3/detection_loss‚0.9803513174335803 .loss/FBt4/detection_loss‚1.795008470614751 _step‚162 % _timestamp‚1.7885314716140747e+09 /loss/FRt1/assignment_loss‚3.975756260363952 /loss/FRt4/assignment_loss‚4.611963056601011 .loss/FRt1/detection_loss‚2.179767976636472úÁ3ë]£× 0loss/FBt1/assignment_loss‚0.1836220922527616 /loss/FRt2/detection_loss‚2.9859500098973513 1loss/SRqqt3/detection_loss‚0.9803513174335803 .loss/FBt4/detection_loss‚1.795008470614751 _step‚162 % _timestamp‚1.7885314716140747e+09 /loss/FRt1/assignment_loss‚3.975756260363952 /loss/FRt4/assignment_loss‚4.611963056601011 .loss/FRt1/detection_loss‚2.179767976636472 1loss/SRqqt1/detection_loss‚1.9138377867639065 2loss/SRqqt2/assignment_loss‚1.8324582850933075 2loss/SRqqt4/assignment_loss‚1.7864960005337542 0loss/FBt4/assignment_loss‚0.2132400894024651 epoch‚1 0loss/SRqqt3/assignment_loss‚1.23385250039639 1loss/FBt2/assignment_loss‚0.22755274739377285 /loss/FRt3/detection_loss‚1.5856477964317073 .loss/FRt4/detection_loss‚3.146388734762485 0loss/SRqqt4/detection_loss‚2.781117796897888 .loss/FBt3/detection_loss‚1.040469878539443 &loss/total_loss‚47.247168311613876 trainer/global_step‚1999 2loss/SRqqt1/assignment_loss‚1.6315442897379397 /loss/FRt2/assignment_loss‚4.105194061994553 0loss/FRt3/assignment_loss‚1.8391238918175568 3loss/FBt3/assignment_loss‚8.702166780949483e-07 .loss/FBt1/detection_loss‚2.382553704082966 _runtime‚2455.768203697 1loss/SRqqt2/detection_loss‚2.5471259260177614 .loss/FBt2/detection_loss‚2.268144767731428¢¤>m«¤‚(:„ “¦ëÔßË“– gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.57' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.714 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷œ*Æp¥‚(:å “¦ëÔ¢¸›" disk.dm-0.out‚47535.9462890625 disk.dm-1.in‚0 disk.dm-1.out‚ 78.7734375 disk.drbd1260.in‚0 disk.sda2.out‚0 cpu‚25.259777208765787 proc.cpu.threads‚17 network.recv‚833573 disk.sda1.in‚0 disk.sda.out‚ 78.7734375 disk.sda3.in‚0$ memory_percent‚90.34099578857422& proc.memory.availableMB‚ 1582.53125# proc.memory.rssMB‚ 3020.06640625* proc.memory.percent‚18.433022499084473 network.sent‚ 3.858459e+06) disk./.usagePercent‚79.00748689937765% disk./.usageGB‚1177.2686500549316 disk.sda1.out‚0% disk.nvme0n1.out‚47535.9462890625 disk.sda3.out‚ 78.7734375' disk.nvme0c0n1.out‚47535.9462890625 disk.dm-0.in‚ 5504.4453125 disk.drbd1260.out‚ 0.33984375 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 5504.4453125" disk.nvme0c0n1.in‚ 5504.4453125žºª‘¦‚(:† ¢¦ëÔ™£ß” gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.964' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1928 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w‚° r§‚(:ç ¢¦ëÔ²‘û–# proc.memory.rssMB‚ 3020.06640625 proc.cpu.threads‚17 network.recv‚838754) disk./.usagePercent‚79.00748561933997 disk.drbd1260.out‚0.34375 disk.sda.out‚ 79.17578125 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 5632.453125( proc.memory.availableMB‚ 1582.7734375 cpu‚25.258816630072406 network.sent‚ 3.967249e+06 disk.drbd1260.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 5632.453125& disk.nvme0n1.out‚47753.55322265625 disk.sda3.out‚ 79.17578125* proc.memory.percent‚18.433022499084473 disk.sda2.in‚0( disk.nvme0c0n1.out‚47753.55322265625 disk.dm-1.in‚0 disk.dm-1.out‚ 79.17578125 disk.sda1.in‚0$ memory_percent‚90.33951759338379% disk./.usageGB‚1177.2686309814453 disk.sda2.out‚0 disk.dm-0.in‚ 5632.453125# disk.dm-0.out‚47753.55322265625 disk.sda1.out‚0SÖfœ¨‚(:‘ ±¦ëÔÍßÉ• gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.362' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.272399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í“s)a©‚(:Ö ±¦ëÔ”¶ü˜ disk.sda.in‚0& disk.nvme0n1.out‚47921.15869140625 disk.sda3.out‚79.75 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3020.06640625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚ 5632.453125# disk.dm-0.out‚47920.28369140625 disk.dm-1.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1582.51953125 cpu‚25.257931169599836 network.sent‚ 3.971934e+06% disk./.usageGB‚1177.2686042785645 disk.sda2.out‚0 disk.drbd1260.out‚0.34375$ memory_percent‚90.34106731414795* proc.memory.percent‚18.433022499084473 network.recv‚843713 disk.sda.out‚79.75 disk.nvme0n1.in‚ 5632.453125 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 5632.453125( disk.nvme0c0n1.out‚47920.90869140625 disk.dm-1.out‚79.75 disk.sda1.out‚0) disk./.usagePercent‚79.00748382728725jó•a‘ª‚(:† À¦ëÔŽŒ— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.156' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4312 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Êi]ªh«‚(:Ý À¦ëÔΞêš disk.nvme0n1.in‚ 5632.453125! disk.nvme0c0n1.in‚ 5632.453125 disk.drbd1260.in‚0' proc.memory.availableMB‚ 1582.765625# proc.memory.rssMB‚ 3020.06640625 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 48348.0234375 disk.dm-1.in‚0 disk.dm-1.out‚ 80.44140625$ memory_percent‚90.33956527709961* proc.memory.percent‚18.433022499084473 cpu‚25.257059728001526 proc.cpu.threads‚17 network.sent‚ 3.976751e+06) disk./.usagePercent‚79.00612109918248% disk./.usageGB‚1177.2482986450195 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 80.44140625 disk.dm-0.in‚ 5632.453125 disk.dm-0.out‚ 48348.0234375 network.recv‚847017" disk.nvme0n1.out‚ 48348.0234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.34765625 disk.sda.in‚0 disk.sda.out‚ 80.44140625*t›ñ2¬- _wandbruntime‚2514  _runtime‚2514ìM°-Ç­Á & lr-AdamW/pg1‚0.0008046619079510855 & lr-AdamW/pg2‚0.0008046619079510855 trainer/global_step‚2049 % _timestamp‚1.7885315305323775e+09 _runtime‚2514.686123184 _step‚163ÏèZÌ®Æ & lr-AdamW/pg1‚0.0008046619079510855 & lr-AdamW/pg2‚0.0008046619079510855 trainer/global_step‚2049 % _timestamp‚1.7885315305323775e+09 _runtime‚2514.686123184 _step‚163£^»2¯- _wandbruntime‚2514  _runtime‚2514Šœ½$Ç'°ÁO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4055641780736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55433338 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37497771 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86503888897536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82448247116800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86504140568064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87382028536320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78561096627216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4127143689728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286426112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310366208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4055875448320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38445307 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14608987 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82605234862117 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚78561079587856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4044384255337 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86503888897536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22888784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4055641780736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38445307 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55406996 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22888786 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4126857263616 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91509196166144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38445305 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55406996 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91508885799936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82605480882553 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚164 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4055875448320 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82448247116800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 trainer/global_step‚2049 _runtime‚2514.687637439 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86504140568064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82448265119744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4044155274261 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38445305 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14609654 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4044155274261 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38445305 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14609654 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55433338 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37497771 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86503888897536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82448247116800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86504140568064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87382028536320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78561096627216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4127143689728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286426112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310366208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4055875448320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38445307 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14608987 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82605234862117 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚78561079587856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4044384255337 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86503888897536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22888784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4055641780736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38445307 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55406996 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22888786 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4126857263616 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91509196166144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38445305 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55406996 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91508885799936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82605480882553 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚164 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4055875448320¤q ‘O2²- _wandbruntime‚2515  _runtime‚2515Cñ«*` ³ÌM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78598626984640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4057605806592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82487551156736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38463580 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14615694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55460056 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55433688 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86545390644224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82487533153792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55433688 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82487533153792 M>DeviceStatsMonitor±*`žr.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38463582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87423712506368 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885315316642327e+09 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37515280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93897268 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91552561330176 _runtime‚2515.818587141 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93923638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4057605806592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91552880071680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4057839487488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4128848823808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4046113763247 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87423736446464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55460056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129143625216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38463580 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 trainer/global_step‚2049 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93897268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4046342744531 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37515972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82487551156736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ô²×&´ÑM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885315316642327e+09 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37515280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93897268 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91552561330176 _runtime‚2515.818587141 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93923638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4057605806592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91552880071680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4057839487488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4128848823808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4046113763247 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87423736446464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55460056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129143625216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38463580 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 trainer/global_step‚2049 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93897268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4046342744531 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37515972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82487551156736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78598626984640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4057605806592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82487551156736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38463580 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14615694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55460056 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55433688 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86545390644224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82487533153792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55433688 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82487533153792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38463582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87423712506368 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582¥ŠáûU2µ- _wandbruntime‚2515  _runtime‚2515³”à3X¶Ò /loss/FRt2/assignment_loss‚4.164532788097858 1loss/SRqqt1/assignment_loss‚1.768722690782457 2loss/SRqqt4/assignment_loss‚2.1421403678563924 /loss/SRqqt2/detection_loss‚2.42939330637455 2loss/SRqqt3/assignment_loss‚0.8861133126859985 _step‚166 /loss/FBt3/detection_loss‚0.8505705386226954 .loss/FRt3/assignment_loss‚1.56214423650919 4loss/FBt3/assignment_loss‚0.00010468207602930623 0loss/FBt4/assignment_loss‚0.3414182028652789 /loss/FRt3/detection_loss‚1.2826826802000058 1loss/SRqqt1/detection_loss‚1.9378984514272437 _runtime‚2515.821187793 0loss/FRt1/assignment_loss‚3.7317035574662056 0loss/FBt2/assignment_loss‚0.3818773204692939 /loss/FRt1/detection_loss‚2.4438283129742273 /loss/FBt2/detection_loss‚2.4235295918252735 .loss/FBt4/detection_loss‚1.439584788360766 %loss/total_loss‚45.43917731552926 2loss/SRqqt2/assignment_loss‚1.5337908069292705 0loss/FBt1/assignment_loss‚0.5104151466044381 /loss/FRt2/detection_loss‚2.7895273622125387 .loss/FRt4/detection_loss‚2.926730912307213 .loss/FBt1/detection_loss‚2.081550828048161 epoch‚1 /loss/FRt4/assignment_loss‚4.199286754789023 1loss/SRqqt3/detection_loss‚1.0800434673523032 0loss/SRqqt4/detection_loss‚2.531587208692844 trainer/global_step‚2049 % _timestamp‚1.7885315316677182e+09õÜÍ]·× 2loss/SRqqt2/assignment_loss‚1.5337908069292705 0loss/FBt1/assignment_loss‚0.5104151466044381 /loss/FRt2/detection_loss‚2.7895273622125387 .loss/FRt4/detection_loss‚2.926730912307213 .loss/FBt1/detection_loss‚2.081550828048161 epoch‚1 /loss/FRt4/assignment_loss‚4.199286754789023 1loss/SRqqt3/detection_loss‚1.0800434673523032 0loss/SRqqt4/detection_loss‚2.531587208692844 trainer/global_step‚2049 % _timestamp‚1.7885315316677182e+09 /loss/FRt2/assignment_loss‚4.164532788097858 1loss/SRqqt1/assignment_loss‚1.768722690782457 2loss/SRqqt4/assignment_loss‚2.1421403678563924 /loss/SRqqt2/detection_loss‚2.42939330637455 2loss/SRqqt3/assignment_loss‚0.8861133126859985 _step‚166 /loss/FBt3/detection_loss‚0.8505705386226954 .loss/FRt3/assignment_loss‚1.56214423650919 4loss/FBt3/assignment_loss‚0.00010468207602930623 0loss/FBt4/assignment_loss‚0.3414182028652789 /loss/FRt3/detection_loss‚1.2826826802000058 1loss/SRqqt1/detection_loss‚1.9378984514272437 _runtime‚2515.821187793 0loss/FRt1/assignment_loss‚3.7317035574662056 0loss/FBt2/assignment_loss‚0.3818773204692939 /loss/FRt1/detection_loss‚2.4438283129742273 /loss/FBt2/detection_loss‚2.4235295918252735 .loss/FBt4/detection_loss‚1.439584788360766 %loss/total_loss‚45.43917731552926¦¨£œ¸‚(:‘ ϦëÔ¿ó‚– gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚135.205' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.041000000000004 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iù‹l¹‚(:á ϦëÔ‘è› disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.34765625 disk.sda.in‚0 disk.sda.out‚ 80.94921875 disk.sda2.in‚0 disk.nvme0n1.in‚ 5632.453125 network.sent‚ 3.982895e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 80.94921875% disk.nvme0c0n1.out‚48651.86328125$ memory_percent‚90.34430980682373* proc.memory.percent‚18.433022499084473) disk./.usagePercent‚79.00769733757079 disk.dm-0.in‚ 5632.453125! disk.nvme0c0n1.in‚ 5632.453125) proc.memory.availableMB‚ 1581.98828125 cpu‚25.256117829941473 proc.cpu.threads‚17 network.recv‚851530 disk.dm-0.out‚48651.86328125 disk.dm-1.out‚ 80.94921875 disk.sda2.out‚0# disk.nvme0n1.out‚48651.86328125# proc.memory.rssMB‚ 3020.06640625$ disk./.usageGB‚1177.2717857360849¿‘º‚(:† Þ¦ëÔÀô– gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ÚÁ(q»‚(:æ Þ¦ëÔ”ïãœ# proc.memory.rssMB‚ 3020.06640625% disk./.usageGB‚1177.2717666625977 disk.sda2.in‚0 cpu‚25.255284897489297 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚49149.20654296875 disk.sda3.out‚ 81.44921875( disk.nvme0c0n1.out‚49149.20654296875) proc.memory.availableMB‚ 1582.73046875* proc.memory.percent‚18.433022499084473 network.recv‚856960 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3515625 disk.sda.out‚ 81.44921875 disk.nvme0n1.in‚ 5753.21875 disk.sda3.in‚0 disk.dm-0.in‚ 5753.21875 network.sent‚ 4.091881e+06) disk./.usagePercent‚79.00769605753312 disk.dm-1.out‚ 81.44921875 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 5753.21875# disk.dm-0.out‚49149.20654296875# memory_percent‚90.3397798538208¨×èÍš¼‚(: í¦ëÔﺘ• gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚96.919' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.383799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÓ• y½‚(:î í¦ëÔÌ”ï—( disk.nvme0c0n1.out‚49362.42919921875 disk.sda1.out‚0# proc.memory.rssMB‚ 3020.06640625 network.sent‚ 4.096465e+06 network.recv‚861898 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 5819.50390625# disk.dm-0.out‚49362.42919921875 disk.dm-1.out‚ 81.94140625 disk.sda1.in‚0) disk./.usagePercent‚79.00766661666668 disk.sda.out‚ 81.94140625# disk.nvme0c0n1.in‚ 5819.50390625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3515625( proc.memory.availableMB‚ 1582.7265625* proc.memory.percent‚18.433022499084473! disk.nvme0n1.in‚ 5819.50390625$ memory_percent‚90.33980369567871 cpu‚25.254482701263107 proc.cpu.threads‚17$ disk./.usageGB‚1177.271327972412 disk.sda.in‚0& disk.nvme0n1.out‚49362.42919921875 disk.sda3.in‚0 disk.sda3.out‚ 81.94140625&ú‘¾‚(:† ü¦ëÔú–ÿ— gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8972 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ Dn¿‚(:ã ü¦ëÔˆ­ë™) disk./.usagePercent‚79.00766661666668$ disk.nvme0n1.out‚49815.044921875 disk.sda3.out‚ 82.390625 disk.dm-0.in‚ 5819.50390625 disk.dm-1.in‚0# proc.memory.rssMB‚ 3020.06640625* proc.memory.percent‚18.433022499084473 cpu‚25.253648133654202$ disk./.usageGB‚1177.271327972412 disk.sda.in‚0 disk.sda.out‚ 82.390625 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1582.72265625 network.recv‚865333! disk.nvme0n1.in‚ 5819.50390625 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 5819.50390625& disk.nvme0c0n1.out‚49815.044921875! disk.dm-0.out‚49815.044921875 disk.drbd1260.out‚ 0.35546875$ memory_percent‚90.33982753753662 network.sent‚ 4.10314e+06 disk.sda2.in‚0 disk.dm-1.out‚ 82.390625 proc.cpu.threads‚17Gûª:2À- _wandbruntime‚2573  _runtime‚2573i™«ÅÁ¿ _runtime‚2573.801755853 _step‚167 % lr-AdamW/pg1‚0.000803205088423997 % lr-AdamW/pg2‚0.000803205088423997 trainer/global_step‚2099 % _timestamp‚1.7885315896481457e+09É%°ÊÂÄ % _timestamp‚1.7885315896481457e+09 _runtime‚2573.801755853 _step‚167 % lr-AdamW/pg1‚0.000803205088423997 % lr-AdamW/pg2‚0.000803205088423997 trainer/global_step‚2099§3Œ3p2Ã- _wandbruntime‚2573  _runtime‚2573ÛR4CÄÂO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84412566969344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14958060 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885315896493268e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39359057 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88566406453760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4154091154944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23428886 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89466227038720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4153857487360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84412548966400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89466250978816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4226929758208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39359055 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93693180737024 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93692855690752 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚80437597459056 trainer/global_step‚2099 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39359055 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4226628652032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39359057 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38386946 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96104003 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _runtime‚2573.803067085 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301106176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4142093314769 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84412548966400 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56744948 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4153857487360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23428884 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88566658124288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14957356 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88566658124288 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38386240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84412566969344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4154091154944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56771290 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84579690773825 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 VGDeviceStatsMonitor.on_train_batch_star¸Ät… t/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39359057 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88566406453760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4154091154944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23428886 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89466227038720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4153857487360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84412548966400 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89466250978816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4226929758208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39359055 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93693180737024 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93692855690752 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚80437597459056 trainer/global_step‚2099 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39359055 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4226628652032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39359057 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38386946 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96104003 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _runtime‚2573.803067085 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301106176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4142093314769 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84412548966400 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56744948 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4153857487360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23428884 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88566658124288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14957356 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88566658124288 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38386240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84412566969344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4154091154944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56771290 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84579690773825 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96148970 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38405141 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4156055194112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39377330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93736850474496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84619179620235 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39377330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4156055194112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96175340 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38404473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4144051803755 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88607656516608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14965453 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84619425640879 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚2574.751177752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4155821513216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23439686 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56798008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14964787 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282218496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80475127816480 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96148970 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4228915525632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56798008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4155821513216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158592 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚2099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56771640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84451853006336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84451835003392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39377332 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84451853006336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89507911008768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4228633307136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93736544315904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80475144855840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4144280785039 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496À«fˆ&ÈÐM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39377332 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84451853006336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89507911008768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4228633307136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93736544315904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80475144855840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4144280785039 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96175340 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88607908200448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96148970 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38405141 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4156055194112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39377330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93736850474496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84619179620235 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39377330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4156055194112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96175340 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38404473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4144051803755 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88607656516608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14965453 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84619425640879 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚2574.751177752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4155821513216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23439686 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56798008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14964787 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282218496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80475127816480 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96148970 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4228915525632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56798008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4155821513216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158592 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚2099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56771640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84451853006336 YFDeviceStatsMonR”Ì(Nitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84451835003392©zŠ5k2É- _wandbruntime‚2574  _runtime‚2574) Äþ[ÊÕ /loss/FRt1/assignment_loss‚4.244929166940542 /loss/FRt4/assignment_loss‚3.831237288082347 2loss/SRqqt2/assignment_loss‚1.9069649267655153 /loss/FBt1/detection_loss‚2.0530363579591113 .loss/FBt2/detection_loss‚2.489919990301132 epoch‚1 _runtime‚2574.752957727 2loss/SRqqt4/assignment_loss‚1.6278042235860117 0loss/SRqqt1/detection_loss‚2.078477961837121 trainer/global_step‚2099 /loss/FRt3/detection_loss‚1.5002121622077165 /loss/FRt4/detection_loss‚3.5849839273621056 % _timestamp‚1.7885315905995998e+09 1loss/FBt4/assignment_loss‚0.24387695056612704 1loss/FBt2/assignment_loss‚0.27817919027542565 /loss/FBt4/detection_loss‚1.7426681956276298 _step‚170 /loss/FRt2/assignment_loss‚3.906238635381063 2loss/SRqqt1/assignment_loss‚1.6807030362059485 1loss/SRqqt3/detection_loss‚0.9254711658154663 2loss/SRqqt3/assignment_loss‚1.0834305444634273 0loss/FBt1/assignment_loss‚0.4577342584305976 1loss/FBt3/assignment_loss‚0.01328731399098961 -loss/FRt2/detection_loss‚2.73928202688694 1loss/SRqqt2/detection_loss‚2.4947365545309506 /loss/FBt3/detection_loss‚0.9739525453784527 %loss/total_loss‚46.52567011857359 0loss/FRt3/assignment_loss‚1.8017808512235298 /loss/FRt1/detection_loss‚2.1755736638338137 1loss/SRqqt4/detection_loss‚2.6911891809216253ûRM`ËÚ /loss/FRt1/assignment_loss‚4.244929166940542 /loss/FRt4/assignment_loss‚3.831237288082347 2loss/SRqqt2/assignment_loss‚1.9069649267655153 /loss/FBt1/detection_loss‚2.0530363579591113 .loss/FBt2/detection_loss‚2.489919990301132 epoch‚1 _runtime‚2574.752957727 2loss/SRqqt4/assignment_loss‚1.6278042235860117 0loss/SRqqt1/detection_loss‚2.078477961837121 trainer/global_step‚2099 /loss/FRt3/detection_loss‚1.5002121622077165 /loss/FRt4/detection_loss‚3.5849839273621056 % _timestamp‚1.7885315905995998e+09 1loss/FBt4/assignment_loss‚0.24387695056612704 1loss/FBt2/assignment_loss‚0.27817919027542565 /loss/FBt4/detection_loss‚1.7426681956276298 _step‚170 /loss/FRt2/assignment_loss‚3.906238635381063 2loss/SRqqt1/assignment_loss‚1.6807030362059485 1loss/SRqqt3/detection_loss‚0.9254711658154663 2loss/SRqqt3/assignment_loss‚1.0834305444634273 0loss/FBt1/assignment_loss‚0.4577342584305976 1loss/FBt3/assignment_loss‚0.01328731399098961 -loss/FRt2/detection_loss‚2.73928202688694 1loss/SRqqt2/detection_loss‚2.4947365545309506 /loss/FBt3/detection_loss‚0.9739525453784527 %loss/total_loss‚46.52567011857359 0loss/FRt3/assignment_loss‚1.8017808512235298 /loss/FRt1/detection_loss‚2.1755736638338137 1loss/SRqqt4/detection_loss‚2.6911891809216253ª/ŒiœÌ‚(:‘ ‹§ëÔðˆÞ— gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.146' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.829199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨p oÍ‚(:ä ‹§ëÔ¾­¹˜* proc.memory.percent‚18.433046340942383 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 5819.50390625 proc.cpu.threads‚17 network.recv‚869835 disk.sda1.out‚0 disk.sda.out‚ 82.8203125$ disk.nvme0n1.out‚49904.919921875 disk.sda3.out‚ 82.8203125& disk.nvme0c0n1.out‚49904.919921875" memory_percent‚90.340256690979) proc.memory.availableMB‚ 1582.65234375) disk./.usagePercent‚79.00768428118656% disk./.usageGB‚1177.2715911865234 disk.dm-1.out‚ 82.8203125 disk.drbd1260.out‚ 0.35546875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 5819.50390625" proc.memory.rssMB‚ 3020.0703125 cpu‚25.25260376001033 network.sent‚ 4.107448e+06 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 5819.50390625! disk.dm-0.out‚49904.919921875ñR#šÎ‚(:… š§ëÔ͈ – gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0046 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ó˜ÄtÏ‚(:é š§ëÔ¢Çï™ disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 83.5546875( disk.nvme0c0n1.out‚50261.35888671875# disk.dm-0.out‚50261.35888671875( proc.memory.availableMB‚ 1582.3984375" proc.memory.rssMB‚ 3020.0703125* proc.memory.percent‚18.433046340942383 cpu‚25.251895435062014 network.sent‚ 4.214337e+06 network.recv‚875216$ disk./.usageGB‚1177.248104095459 disk.dm-1.in‚0 disk.drbd1260.out‚0.359375 disk.sda.in‚0 disk.sda.out‚ 83.5546875! disk.nvme0n1.in‚ 5819.50390625# disk.nvme0c0n1.in‚ 5819.50390625 disk.dm-0.in‚ 5819.50390625 disk.sda1.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚50261.35888671875$ memory_percent‚90.34180641174316 proc.cpu.threads‚17) disk./.usagePercent‚79.00610804279823 disk.sda3.in‚0 disk.dm-1.out‚ 83.5546875x  ™‘Ђ(:† ©§ëÔÔ˜å• gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'iãuÑ‚(:ê ©§ëÔ§“´˜# disk.dm-0.out‚50450.56982421875 disk.dm-1.in‚0 disk.nvme0n1.in‚ 5966.3203125 disk.sda3.in‚0* proc.memory.percent‚18.433046340942383 proc.cpu.threads‚17 network.sent‚ 4.220829e+06 disk.drbd1260.out‚0.359375( proc.memory.availableMB‚ 1582.6328125" proc.memory.rssMB‚ 3020.0703125 disk.dm-1.out‚ 84.08984375 disk.sda1.in‚0 network.recv‚880218% disk./.usageGB‚1177.1139450073242" disk.nvme0c0n1.in‚ 5966.3203125 disk.dm-0.in‚ 5966.3203125 disk.sda.in‚0 disk.sda.out‚ 84.08984375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚50450.56982421875 disk.sda3.out‚ 84.08984375$ memory_percent‚90.34037590026855 cpu‚25.251064217056772) disk./.usagePercent‚78.99710451382653( disk.nvme0c0n1.out‚50450.56982421875å‹Ó œÒ‚(:‘ ¸§ëÔÛ‘´– gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚140.096' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.019199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 w§oÓ‚(:ä ¸§ëÔÞ•†— network.sent‚ 4.225694e+06% disk./.usageGB‚1176.7113800048828 disk.sda1.in‚0 disk.nvme0n1.in‚ 5966.3203125% disk.nvme0n1.out‚50785.2724609375 disk.sda3.in‚0 disk.dm-0.in‚ 5966.3203125" disk.dm-0.out‚50785.2724609375) disk./.usagePercent‚78.97008803873817 disk.sda1.out‚0 disk.sda3.out‚ 84.5703125' proc.memory.availableMB‚ 1582.390625" proc.memory.rssMB‚ 3020.0703125* proc.memory.percent‚18.433046340942383 network.recv‚883558 disk.dm-1.in‚0 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 5966.3203125' disk.nvme0c0n1.out‚50785.2724609375$ memory_percent‚90.34185409545898 cpu‚25.25023866437051 proc.cpu.threads‚17 disk.dm-1.out‚ 84.5703125 disk.drbd1260.out‚ 0.36328125 disk.sda.in‚0 disk.sda.out‚ 84.5703125 disk.sda2.in‚0 disk.sda2.out‚0CM¿2Ô- _wandbruntime‚2633  _runtime‚2633(ˆÈÄÕ¾ % lr-AdamW/pg1‚0.000801665972447258 % lr-AdamW/pg2‚0.000801665972447258 trainer/global_step‚2149 $ _timestamp‚1.788531649163191e+09 _runtime‚2633.316770699 _step‚171 iáÉÖà % lr-AdamW/pg2‚0.000801665972447258 trainer/global_step‚2149 $ _timestamp‚1.788531649163191e+09 _runtime‚2633.316770699 _step‚171 % lr-AdamW/pg1‚0.000801665972447258«g4Ôõ2×- _wandbruntime‚2633  _runtime‚2633'~”ÝÈ'ØÂO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4252073193984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4252073193984 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39275675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23968986 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91550425541120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95877098739200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4240260336353 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4252306861568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4326649257984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40272807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82314132369616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90628924009984 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86376850816000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58109242 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86554146685533 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40272805 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58082900 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95876784178688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15306689 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58082900 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23968984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82314115330256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39275001 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86554392705969 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98355705 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90629175680512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4240031355277 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86376868818944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90628924009984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91550449481216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885316491642985e+09 _step‚172 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86376850816000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15306017 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚2633.318004825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58109242 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98382049 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23968984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82314115330256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39275001 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86554392705969 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98355705 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90629175680512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4240031355277 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86376868818944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90628924009984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91550449481216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885316491642985e+09 _step‚172 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86376850816000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15306017 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚2633.318004825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58109242 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98382049 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4252073193984 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39275675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23968986 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91550425541120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95877098739200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4240260336353 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4252306861568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4326649257984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40272807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82314132369616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90628924009984 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86376850816000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58109242 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86554146685533 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40272805 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58082900¬Õó¸ô2Ú- _wandbruntime‚2634  _runtime‚2634ˆÃÑ¡aÛÊM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚90670425756672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚90670174072832 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40291082 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58109592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2634.299380821 _step‚173 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40291082 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2149 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15314080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95920460704256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4254270900736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58135960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39293868 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98427042 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86416136852992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82351645687680 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58135960 XJDeviceStatsMonito2üO¼or.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98427042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86593635531943 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23979788 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39293145 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98400672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86416154855936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40291080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98400672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4242218825547 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496ÁýÕ&ÜÏM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚90670425756672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚90670174072832 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40291082 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58109592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2634.299380821 _step‚173 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40291082 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2149 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15314080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95920460704256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4254270900736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58135960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39293868 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98427042 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86416136852992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82351645687680 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58135960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98427042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86593635531943 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23979788 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39293145 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98400672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86416154855936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40291080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98400672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4242218825547 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496­.2Òî2Ý- _wandbruntime‚2634  _runtime‚2634SG‚VÞÐ /loss/FBt3/detection_loss‚0.8656977746333625 /loss/FBt4/detection_loss‚1.4173930727622726 $ _timestamp‚1.788531650146999e+09 /loss/FRt2/detection_loss‚3.1223497824235396 3loss/FBt3/assignment_loss‚8.505231333972492e-05 .loss/FRt4/detection_loss‚3.436451256275177 0loss/SRqqt1/detection_loss‚2.010222220172485 2loss/SRqqt4/assignment_loss‚2.9556805064280827 /loss/FBt1/detection_loss‚2.5375998675823213 /loss/FBt2/detection_loss‚2.5356787939866385 /loss/FRt1/detection_loss‚2.4850967259783494 /loss/FRt3/assignment_loss‚1.541748062356726 2loss/SRqqt1/assignment_loss‚1.5752997227633994 0loss/SRqqt2/assignment_loss‚1.94627597140005 epoch‚1 2loss/SRqqt3/assignment_loss‚1.0967531111639213 1loss/FBt1/assignment_loss‚0.38638719819524947 0loss/SRqqt4/detection_loss‚2.604560498396556 trainer/global_step‚2149 _step‚174 0loss/FRt2/assignment_loss‚4.3624372048811475 ,loss/FRt3/detection_loss‚1.5924569512536 $loss/total_loss‚48.0367844693218 _runtime‚ 2634.30076538 1loss/FBt4/assignment_loss‚0.22935960678004685 /loss/FRt1/assignment_loss‚2.963711729175166 /loss/FRt4/assignment_loss‚4.357540893554687 1loss/SRqqt3/detection_loss‚1.0416167097202702 0loss/FBt2/assignment_loss‚0.2820258991479123 1loss/SRqqt2/detection_loss‚2.6903558579775004Šú5[ßÕ 0loss/FBt2/assignment_loss‚0.2820258991479123 1loss/SRqqt2/detection_loss‚2.6903558579775004 /loss/FBt3/detection_loss‚0.8656977746333625 /loss/FBt4/detection_loss‚1.4173930727622726 $ _timestamp‚1.788531650146999e+09 /loss/FRt2/detection_loss‚3.1223497824235396 3loss/FBt3/assignment_loss‚8.505231333972492e-05 .loss/FRt4/detection_loss‚3.436451256275177 0loss/SRqqt1/detection_loss‚2.010222220172485 2loss/SRqqt4/assignment_loss‚2.9556805064280827 /loss/FBt1/detection_loss‚2.5375998675823213 /loss/FBt2/detection_loss‚2.5356787939866385 /loss/FRt1/detection_loss‚2.4850967259783494 /loss/FRt3/assignment_loss‚1.541748062356726 2loss/SRqqt1/assignment_loss‚1.5752997227633994 0loss/SRqqt2/assignment_loss‚1.94627597140005 epoch‚1 2loss/SRqqt3/assignment_loss‚1.0967531111639213 1loss/FBt1/assignment_loss‚0.38638719819524947 0loss/SRqqt4/detection_loss‚2.604560498396556 trainer/global_step‚2149 _step‚174 0loss/FRt2/assignment_loss‚4.3624372048811475 ,loss/FRt3/detection_loss‚1.5924569512536 $loss/total_loss‚48.0367844693218 _runtime‚ 2634.30076538 1loss/FBt4/assignment_loss‚0.22935960678004685 /loss/FRt1/assignment_loss‚2.963711729175166 /loss/FRt4/assignment_loss‚4.357540893554687 1loss/SRqqt3/detection_loss‚1.0416167097202702® ߯‘à‚(:† ǧëÔ€»¼— gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9904 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±!šnrá‚(:ç ǧëÔš˜¾˜ disk.sda3.out‚ 84.91015625" disk.nvme0c0n1.in‚ 5966.3203125 network.sent‚ 4.231697e+06 disk.sda2.in‚0 disk.nvme0n1.in‚ 5966.3203125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚90.34361839294434 disk.sda3.in‚0' disk.nvme0c0n1.out‚51102.1005859375" disk.dm-0.out‚51102.1005859375 disk.dm-1.out‚ 84.91015625 disk.drbd1260.out‚ 0.36328125( proc.memory.availableMB‚ 1582.1015625" proc.memory.rssMB‚ 3020.0703125* proc.memory.percent‚18.433046340942383 disk.sda.out‚ 84.91015625 disk.dm-0.in‚ 5966.3203125 cpu‚25.24936810496906 proc.cpu.threads‚17 network.recv‚888252( disk./.usagePercent‚78.9700905988135% disk./.usageGB‚1176.7114181518555 disk.sda.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚51102.1005859375ë"B­‘â‚(:† Ö§ëÔ뫺— gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7334 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+¹·Jpã‚(:å Ö§ëÔØ؈œ proc.cpu.threads‚17$ disk./.usageGB‚1176.711368560791 disk.dm-1.in‚0 disk.dm-1.out‚ 85.3515625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda3.out‚ 85.3515625) proc.memory.availableMB‚ 1582.34765625) disk./.usagePercent‚78.97008727071557( disk.nvme0c0n1.out‚51433.70361328125# disk.dm-0.out‚51433.70361328125 disk.sda1.in‚0 disk.drbd1260.out‚ 0.3671875 disk.sda.out‚ 85.3515625 disk.sda2.out‚0" proc.memory.rssMB‚ 3020.0703125 network.sent‚ 4.338681e+06 network.recv‚893250& disk.nvme0n1.out‚51433.70361328125* proc.memory.percent‚18.433046340942383 cpu‚25.24855930732405" disk.nvme0c0n1.in‚ 5966.3203125 disk.dm-0.in‚ 5966.3203125 disk.sda2.in‚0 disk.nvme0n1.in‚ 5966.3203125 disk.sda3.in‚0" memory_percent‚90.342116355896“«Ú|ä‚(:… å§ëÔ×ß¡— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.981' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7962 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªÕkvå‚(:ë å§ëÔ§÷Æš disk.dm-1.out‚ 85.75390625* proc.memory.percent‚18.433046340942383) disk./.usagePercent‚78.97008343060254! disk.nvme0n1.in‚ 6132.58203125 disk.sda3.out‚ 85.75390625 disk.dm-0.in‚ 6132.58203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.3671875" memory_percent‚90.342116355896$ disk./.usageGB‚1176.711311340332 disk.sda.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 6132.58203125( disk.nvme0c0n1.out‚51582.24658203125 disk.dm-1.in‚0) proc.memory.availableMB‚ 1582.34765625" proc.memory.rssMB‚ 3020.0703125 network.sent‚ 4.345101e+06 disk.sda2.in‚0& disk.nvme0n1.out‚51582.24658203125 disk.sda3.in‚0# disk.dm-0.out‚51582.24658203125 disk.drbd1260.in‚0 cpu‚25.24767290503043 proc.cpu.threads‚17 network.recv‚898078 disk.sda.out‚ 85.75390625.v-Qœæ‚(:‘ ô§ëÔ¤º– gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚140.466' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.093200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÎÊtç‚(:é ô§ëÔ£‡°—* proc.memory.percent‚18.433046340942383 proc.cpu.threads‚17) disk./.usagePercent‚78.96330486311093 disk.dm-0.in‚ 6132.58203125! disk.dm-0.out‚51770.462890625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 86.19140625 network.recv‚901260% disk./.usageGB‚1176.6103057861328 disk.sda2.in‚0$ disk.nvme0n1.out‚51770.462890625 disk.sda3.out‚ 86.19140625# disk.nvme0c0n1.in‚ 6132.58203125& disk.nvme0c0n1.out‚51770.462890625 cpu‚25.246827822077464 network.sent‚ 4.351928e+06 disk.dm-1.out‚ 86.19140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.37109375 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚90.34209251403809( proc.memory.availableMB‚ 1582.3515625 disk.sda1.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 6132.58203125" proc.memory.rssMB‚ 3020.0703125:á062è- _wandbruntime‚2689  _runtime‚2689 ¾Äé¾ _step‚175 % lr-AdamW/pg1‚0.000800044880770331 % lr-AdamW/pg2‚0.000800044880770331 trainer/global_step‚2199 $ _timestamp‚1.788531705531988e+09 _runtime‚2689.685534883>ëéÔÉêà $ _timestamp‚1.788531705531988e+09 _runtime‚2689.685534883 _step‚175 % lr-AdamW/pg1‚0.000800044880770331 % lr-AdamW/pg2‚0.000800044880770331 trainer/global_step‚2199¯N–©|2ë- _wandbruntime‚2689  _runtime‚2689Ä©ÚEPìÆO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41186557 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59447194 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4350522568192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24509084 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4337969395785 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4426084344320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92691441566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88341170668544 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98060708387840 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92691693236736 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98061025045504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41186555 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚176 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4350522568192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92691693236736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41186555 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59420852 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40166624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338198376861 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88341170668544 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4350288900608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4426377061888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41186557 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_byteúTS™|s.large_pool.freed‚88341152665600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59447194 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41186557 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59447194 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4350522568192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24509084 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4337969395785 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4426084344320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92691441566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88341170668544 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98060708387840 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92691693236736 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98061025045504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41186555 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚176 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4350522568192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92691693236736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41186555 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59420852 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40166624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338198376861 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88341170668544 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4350288900608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4426377061888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41186557 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88341152665600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59447194 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92732691629056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4428368395776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84228163558880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4340156866055 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24519888 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98104700349440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41204832 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41204830 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24519886 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88568091443651 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339927884771 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41204830 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100678744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59473912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚88380456705536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 trainer/global_step‚2199 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88380456705536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352252926464 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98104389996544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93676308013568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100678744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59473912 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40184028 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4428081982976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚2690.777235399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92732943312896 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100652374 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92732943312896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4352486607360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352252926464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582fÕfOÛ&ðÕM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24519886 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88568091443651 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339927884771 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41204830 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100678744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59473912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚88380456705536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 trainer/global_step‚2199 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88380456705536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352252926464 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98104389996544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93676308013568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100678744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59473912 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40184028 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4428081982976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚2690.777235399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92732943312896 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100652374 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92732943312896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4352486607360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352252926464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15664807 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92732691629056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4428368395776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84228163558880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4340156866055 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24519888 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98104700349440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41204832 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41204830 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896±Ř‘‡2ñ- _wandbruntime‚2690  _runtime‚2690»JÛümòÔ 0loss/FRt2/assignment_loss‚3.8994673755433826 2loss/SRqqt2/assignment_loss‚2.1148113161325455 /loss/FBt1/detection_loss‚2.3315867327508473 /loss/FBt2/detection_loss‚2.4165051698684694 trainer/global_step‚2199 /loss/FRt1/detection_loss‚2.8350967708975077 3loss/FBt3/assignment_loss‚0.0042366272148986655 /loss/FRt3/detection_loséó_ís‚1.3703120608151573 0loss/SRqqt4/detection_loss‚2.767971441745758 _step‚178 1loss/FBt1/assignment_loss‚0.29291243970432596 0loss/SRqqt1/detection_loss‚2.218984219431877 % _timestamp‚1.7885317066248972e+09 _runtime‚ 2690.77840307 0loss/FRt1/assignment_loss‚3.2462477944791317 2loss/SRqqt1/assignment_loss‚1.6014748344197869 1loss/SRqqt3/assignment_loss‚1.152996106799824 0loss/FBt2/assignment_loss‚0.2708986572593858 /loss/FRt4/detection_loss‚3.0784596777879276 %loss/total_loss‚46.19408879435065 0loss/FBt4/assignment_loss‚0.2987698593655921 .loss/FRt3/assignment_loss‚1.80718350103221 /loss/FRt4/assignment_loss‚4.167731149838521 0loss/SRqqt2/detection_loss‚2.450460535558787 .loss/FRt2/detection_loss‚3.060114703796528 /loss/FBt3/detection_loss‚0.6866978639057683 epoch‚1 2loss/SRqqt4/assignment_loss‚2.0383493292331694 1loss/SRqqt3/detection_loss‚0.8199919985329851 /loss/FBt4/detection_loss‚1.2628286282362586Õã™_óÙ .loss/FRt3/assignment_loss‚1.80718350103221 /loss/FRt4/assignment_loss‚4.167731149838521 0loss/SRqqt2/detection_loss‚2.450460535558787 .loss/FRt2/detection_loss‚3.060114703796528 /loss/FBt3/detection_loss‚0.6866978639057683 epoch‚1 2loss/SRqqt4/assignment_loss‚2.0383493292331694 1loss/SRqqt3/detection_loss‚0.8199919985329851 /loss/FBt4/detection_loss‚1.2628286282362586 0loss/FRt2/assignment_loss‚3.8994673755433826 2loss/SRqqt2/assignment_loss‚2.1148113161325455 /loss/FBt1/detection_loss‚2.3315867327508473 /loss/FBt2/detection_loss‚2.4165051698684694 trainer/global_step‚2199 /loss/FRt1/detection_loss‚2.8350967708975077 3loss/FBt3/assignment_loss‚0.0042366272148986655 /loss/FRt3/detection_loss‚1.3703120608151573 0loss/SRqqt4/detection_loss‚2.767971441745758 _step‚178 1loss/FBt1/assignment_loss‚0.29291243970432596 0loss/SRqqt1/detection_loss‚2.218984219431877 % _timestamp‚1.7885317066248972e+09 _runtime‚ 2690.77840307 0loss/FRt1/assignment_loss‚3.2462477944791317 2loss/SRqqt1/assignment_loss‚1.6014748344197869 1loss/SRqqt3/assignment_loss‚1.152996106799824 0loss/FBt2/assignment_loss‚0.2708986572593858 /loss/FRt4/detection_loss‚3.0784596777879276 %loss/total_loss‚46.19408879435065 0loss/FBt4/assignment_loss‚0.2987698593655921²ó±‘ô‚(:† ƒ¨ëÔ³õä– gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.526' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3052 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̈OQoõ‚(:ä ƒ¨ëÔôê™ disk.dm-1.out‚ 86.5078125 disk.sda.in‚0 disk.sda3.out‚ 86.5078125 proc.cpu.threads‚17 network.sent‚ 4.355218e+06) disk./.usagePercent‚78.96780854765328% disk.nvme0c0n1.out‚52246.93359375 disk.sda1.in‚0 disk.sda2.in‚0% disk./.usageGB‚1176.6774139404297 disk.drbd1260.in‚0 disk.sda.out‚ 86.5078125! disk.nvme0n1.in‚ 6132.58203125# disk.nvme0n1.out‚52246.93359375) proc.memory.availableMB‚ 1581.98046875* proc.memory.percent‚18.433046340942383 cpu‚25.245975839084217 disk.dm-0.in‚ 6132.58203125 disk.dm-0.out‚52246.93359375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.37109375 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚90.34435749053955" proc.memory.rssMB‚ 3020.0703125 network.recv‚905032# disk.nvme0c0n1.in‚ 6132.58203125 disk.dm-1.in‚0€xLY‘ö‚(:† ’¨ëÔ¿‡— gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç™8új÷‚(:ß ’¨ëÔ¨ˆ› network.recv‚910134 disk.dm-0.in‚ 6132.58203125" disk.dm-0.out‚52485.6318359375 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 87.109375) proc.memory.availableMB‚ 1582.23046875 disk.sda1.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚52485.6318359375# memory_percent‚90.3428316116333" proc.memory.rssMB‚ 3020.0703125* proc.memory.percent‚18.433046340942383 cpu‚25.24524752218074) disk./.usagePercent‚78.97388847058396$ disk./.usageGB‚1176.768009185791# disk.nvme0c0n1.in‚ 6132.58203125' disk.nvme0c0n1.out‚52485.6318359375 disk.dm-1.out‚ 87.109375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 87.109375 network.sent‚ 4.462339e+06 disk.dm-1.in‚0 disk.drbd1260.out‚0.375! disk.nvme0n1.in‚ 6132.58203125 disk.sda3.in‚0 proc.cpu.threads‚17J„^Áø‚(:… ¡¨ëÔ”ᤗ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.548' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.1096 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆë˜Öqù‚(:æ ¡¨ëÔᇔ› disk.sda3.out‚ 87.61328125 disk.dm-1.out‚ 87.61328125 disk.sda1.in‚0" proc.memory.rssMB‚ 3020.0703125 cpu‚25.244638573246377 disk.sda.out‚ 87.61328125 disk.sda2.out‚0# memory_percent‚90.3428316116333) proc.memory.availableMB‚ 1582.23046875* proc.memory.percent‚18.433046340942383 proc.cpu.threads‚17 network.sent‚ 4.468695e+06 network.recv‚914984% disk./.usageGB‚1176.7481956481934! disk.nvme0n1.in‚ 6276.05078125% disk.nvme0n1.out‚52896.8349609375' disk.nvme0c0n1.out‚52896.8349609375 disk.dm-0.in‚ 6276.05078125" disk.dm-0.out‚52896.8349609375( disk./.usagePercent‚78.9725587674511 disk.sda.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6276.05078125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.375 disk.sda3.in‚0 ”û‘ú‚(:† °¨ëÔ‚Ý– gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8204 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼[n±nû‚(:ã °¨ëÔò…ô› disk.sda1.in‚0 disk.sda.out‚ 88.3203125* proc.memory.percent‚18.433046340942383 cpu‚25.243845898029193 network.sent‚ 4.475475e+06) disk./.usagePercent‚78.97413372580175# disk.nvme0n1.out‚53233.79296875# disk.nvme0c0n1.in‚ 6276.05078125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0% disk./.usageGB‚1176.7716636657715! disk.nvme0n1.in‚ 6276.05078125 disk.sda3.out‚ 88.3203125 disk.dm-1.out‚ 88.3203125 disk.drbd1260.out‚ 0.37890625 disk.sda2.out‚0$ memory_percent‚90.34285545349121( proc.memory.availableMB‚ 1582.2265625 disk.sda3.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 3020.0703125 proc.cpu.threads‚17 network.recv‚918181% disk.nvme0c0n1.out‚53233.79296875 disk.dm-0.in‚ 6276.05078125 disk.dm-0.out‚53233.79296875.ÌHa2ü- _wandbruntime‚2748  _runtime‚2748<ñ·SÇýÁ trainer/global_step‚2249 % _timestamp‚1.7885317646866887e+09 _runtime‚2748.840335269 _step‚179 & lr-AdamW/pg1‚0.0007983421512262918 & lr-AdamW/pg2‚0.00079834215122629181{6ÌþÆ % _timestamp‚1.7885317646866887e+09 _runtime‚2748.840335269 _step‚179 & lr-AdamW/pg1‚0.0007983421512262918 & lr-AdamW/pg2‚0.0007983421512262918 trainer/global_step‚2249³Z»Ñ+2ÿ- _wandbruntime‚2748  _runtime‚27486µ5:Î'€ÈO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94753959122432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60758804 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42100307 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41054968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885317646882389e+09 _step‚180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4448504607232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100245005394432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95718822545920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41055644 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25049186 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4436136417369 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90305472518144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102885453 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86067168112016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25049184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16005784 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60758804 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4448738274816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90305454515200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436·=£÷Ó'ÍO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60758804 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4448738274816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90305454515200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94753959122432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60758804 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42100307 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41054968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885317646882389e+09 _step‚180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4448504607232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100245005394432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95718822545920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41055644 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25049186 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4436136417369 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90305472518144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102885453 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86067168112016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25049184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16005784´ªV02‚- _wandbruntime‚2750  _runtime‚27506Ÿ.qƒÒM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102930446 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90344740552192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102904076 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90344758555136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42118582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4450702313984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100288383258112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4527876742144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102930446 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60811864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16013156 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oví!ªJgersize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86104681430080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885317660023828e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25059986 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100288693611008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60785496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86104698469440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42118580 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95760530456064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25059988 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42118580 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90344740552192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60811864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102904076 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95760506515968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4437865925279 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90542547355359 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94795460869120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4450468633088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16012474 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41072460 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚2249×ñÝ&„×M C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42118580 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95760530456064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25059988 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42118580 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90344740552192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60811864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102904076 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95760506515968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4437865925279 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90542547355359 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94795460869120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4450468633088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16012474 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41072460 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚2249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102930446 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90344740552192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102904076 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90344758555136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42118582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4450702313984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100288383258112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4527876742144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102930446 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60811864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16013156 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86104681430080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885317660023828e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25059986 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100288693611008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60785496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86104698469440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944µQ—Z2…- _wandbruntime‚2750  _runtime‚2750`Ó~Æ[†Õ /loss/FBt3/detection_loss‚0.9212075066752732 2loss/SRqqt2/assignment_loss‚2.1162350857257843 /loss/FRt2/detection_loss‚3.6889817854937386 /loss/FBt1/detection_loss‚1.9900204496724265 % _timestamp‚1.7885317660069804e+09 0loss/FRt2/assignment_loss‚3.8522167626549217 0loss/FRt3/assignment_loss‚1.6187984370871593 1loss/FBt4/assignment_loss‚0.39457294672372123 epoch‚1 _step‚182 0loss/SRqqt3/assignment_loss‚1.22198632754664 2loss/SRqqt4/assignment_loss‚2.0726970446109774 .loss/FRt1/detection_loss‚3.107624983010085 0loss/SRqqt4/detection_loss‚2.896080632209778 .loss/FBt4/detection_loss‚1.269515070246487 1loss/FBt2/assignment_loss‚0.30590172064897897 /loss/FRt3/detection_loss‚1.6267077313049843 1loss/SRqqt3/detection_loss‚1.0657541612598385 -loss/FBt2/detection_loss‚2.37354344413394 _runtime‚2750.161044213 0loss/FRt1/assignment_loss‚3.6352414592452673 /loss/FRt4/assignment_loss‚4.008166583145366 /loss/SRqqt2/detection_loss‚2.87719091296196 trainer/global_step‚2249 2loss/SRqqt1/assignment_loss‚1.8073491052074253 1loss/FBt1/assignment_loss‚0.29253208336727504 %loss/total_loss‚47.68784218814898 3loss/FBt3/assignment_loss‚8.514948746096707e-09 /loss/FRt4/detection_loss‚2.8415988149011837 1loss/SRqqt1/detection_loss‚1.7039191318008136X4ÜÆ`‡Ú %loss/total_loss‚47.68784218814898 3loss/FBt3/assignment_loss‚8.514948746096707e-09 /loss/FRt4/detection_loss‚2.8415988149011837 1loss/SRqqt1/detection_loss‚1.7039191318008136 /loss/FBt3/detection_loss‚0.9212075066752732 2loss/SRqqt2/assignment_loss‚2.1162350857257843 /loss/FRt2/detection_loss‚3.6889817854937386 /loss/FBt1/detection_loss‚1.9900204496724265 % _timestamp‚1.7885317660069804e+09 0loss/FRt2/assignment_loss‚3.8522167626549217 0loss/FRt3/assignment_loss‚1.6187984370871593 1loss/FBt4/assignment_loss‚0.39457294672372123 epoch‚1 _step‚182 0loss/SRqqt3/assignment_loss‚1.22198632754664 2loss/SRqqt4/assignment_loss‚2.0726970446109774 .loss/FRt1/detection_loss‚3.107624983010085 0loss/SRqqt4/detection_loss‚2.896080632209778 .loss/FBt4/detection_loss‚1.269515070246487 1loss/FBt2/assignment_loss‚0.30590172064897897 /loss/FRt3/detection_loss‚1.6267077313049843 1loss/SRqqt3/detection_loss‚1.0657541612598385 -loss/FBt2/detection_loss‚2.37354344413394 _runtime‚2750.161044213 0loss/FRt1/assignment_loss‚3.6352414592452673 /loss/FRt4/assignment_loss‚4.008166583145366 /loss/SRqqt2/detection_loss‚2.87719091296196 trainer/global_step‚2249 2loss/SRqqt1/assignment_loss‚1.8073491052074253 1loss/FBt1/assignment_loss‚0.29253208336727504¶Î$à͈‚(:… ¿¨ëÔÑóŠ— gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.835' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.567 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Š ]m‰‚(:â ¿¨ëÔ¦Þý› disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.37890625 disk.sda.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 6276.05078125# disk.nvme0c0n1.in‚ 6276.05078125 disk.dm-0.out‚ 53541.8203125 disk.dm-1.in‚0$ memory_percent‚90.34609794616699* proc.memory.percent‚18.433046340942383% disk./.usageGB‚1176.8345794677734 disk.sda.out‚ 88.78515625$ disk.nvme0c0n1.out‚ 53541.8203125 proc.cpu.threads‚17 disk.sda2.out‚0" disk.nvme0n1.out‚ 53541.8203125 disk.sda3.in‚0 disk.sda3.out‚ 88.78515625 disk.dm-0.in‚ 6276.05078125( proc.memory.availableMB‚ 1581.6953125" proc.memory.rssMB‚ 3020.0703125 cpu‚25.24307265293865 network.sent‚ 4.479682e+06 network.recv‚922672) disk./.usagePercent‚78.97835605806398 disk.dm-1.out‚ 88.78515625 disk.sda1.in‚0ݼ+˜‘Š‚(:† ΨëÔÓúˆ› gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1932 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[0@x‹‚(:í ΨëÔÖÙ½¥" proc.memory.rssMB‚ 3020.0703125* proc.memory.percent‚18.433046340942383& disk.nvme0n1.out‚53587.31591796875 disk.sda3.out‚ 89.25390625 network.sent‚ 4.586749e+06# disk.dm-0.out‚53587.31591796875 disk.sda1.out‚0$ memory_percent‚90.34464359283447 proc.cpu.threads‚17 network.recv‚927742 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 cpu‚25.24230942810821) disk./.usagePercent‚78.98254920546731$ disk./.usageGB‚1176.897060394287 disk.sda.in‚0 disk.sda.out‚ 89.25390625 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 6276.05078125 disk.dm-1.out‚ 89.25390625# disk.nvme0c0n1.in‚ 6276.05078125( disk.nvme0c0n1.out‚53587.31591796875 disk.dm-0.in‚ 6276.05078125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3828125) proc.memory.availableMB‚ 1581.93359375z ™‘Œ‚(:† ݨëÔ–ï²– gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚139.933' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àádp‚(:å ݨëÔ‰““›% proc.memory.availableMB‚ 1582.1875" proc.memory.rssMB‚ 3020.0703125 cpu‚25.24147962096519 proc.cpu.threads‚17 network.recv‚932450 disk.sda3.out‚ 89.72265625( disk.nvme0c0n1.out‚54015.58154296875# disk.dm-0.out‚54015.58154296875* proc.memory.percent‚18.433046340942383 network.sent‚ 4.593157e+06 disk.sda.in‚0 disk.sda.out‚ 89.72265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0$ memory_percent‚90.34309387207031( disk./.usagePercent‚78.9909380603493$ disk./.usageGB‚1177.022060394287 disk.nvme0n1.in‚ 6286.8046875& disk.nvme0n1.out‚54015.58154296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3828125" disk.nvme0c0n1.in‚ 6286.8046875 disk.dm-0.in‚ 6286.8046875 disk.dm-1.out‚ 89.72265625×ÿÌýŽ‚(:… ì¨ëÔ†Þø— gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.277' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6554 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áIZAv‚(:ë ì¨ëÔ©ŠÊ˜% disk./.usageGB‚1176.9985961914062 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.38671875$ memory_percent‚90.34309387207031" proc.memory.rssMB‚ 3020.0703125# disk.dm-0.out‚54364.50244140625 disk.dm-1.out‚ 90.16796875 disk.sda1.in‚0* proc.memory.percent‚18.433046340942383 cpu‚25.24095623756885 network.sent‚ 4.597946e+06# disk.nvme0c0n1.in‚ 6448.38671875( disk.nvme0c0n1.out‚54364.50244140625 disk.dm-0.in‚ 6448.38671875 disk.sda.in‚0 disk.sda.out‚ 90.16796875& disk.nvme0n1.out‚54364.50244140625% proc.memory.availableMB‚ 1582.1875 disk.sda3.in‚0 disk.sda3.out‚ 90.16796875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 6448.38671875 proc.cpu.threads‚17 network.recv‚935897) disk./.usagePercent‚78.98936335800619- Üé2- _wandbruntime‚2808  _runtime‚2808;1³Ç‘Á & lr-AdamW/pg1‚0.0007965581386614252 & lr-AdamW/pg2‚0.0007965581386614252 trainer/global_step‚2299 % _timestamp‚1.7885318242035742e+09 _runtime‚2808.357177812 _step‚183XØ ÎÌ’Æ & lr-AdamW/pg1‚0.0007965581386614252 & lr-AdamW/pg2‚0.0007965581386614252 trainer/global_step‚2299 % _timestamp‚1.7885318242035742e+09 _runtime‚2808.357177812 _step‚183·Y×E£2“- _wandbruntime‚2808  _runtime‚2808•üEãM!”ÈO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96816476678656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62096756 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105110811 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43014055 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43014055 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102429033701888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299009024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4546720313856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41945069 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25589284 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96816476678656 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4625689704448 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105137155 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102428710752768 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4534074457877 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytesê ¡*.all.peak‚ 6966899712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92477514420657 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4625988713472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97803044988416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87943685983216 _step‚184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342Ü¥gÒÓ'•ÍO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43014057 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322949120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92269756364800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4546720313856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87943668943856 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62096756 % _timestamp‚1.7885318242048306e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16355073 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97803021048320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4625689704448 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105137155 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102428710752768 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4534074457877 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92477514420657 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4625988713472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97803044988416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87943685983216 _step‚184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96816476678656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62096756 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105110811 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43014055 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43014055 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102429033701888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299009024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4546720313856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41945069 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25589284 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96816476678656 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4627685936128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97844705018368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41962157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87981199301280 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62149816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4627974446080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97844728958464 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4535803965787 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92517249287711 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4536032947071 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62123448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92309042401792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16362743 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 % _timestamp‚1.7885318253864195e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4548684339712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182148 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92517003267067 _runtime‚2809.540821322 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43032332 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102472390954496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105182148 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62123448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43032330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96857726741504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97844728958464 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4535803965787 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92517249287711 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4536032947071 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62123448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92309042401792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16362743 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 % _timestamp‚1.7885318253864195e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4548684339712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182148 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92517003267067 _runtime‚2809.540821322 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43032332 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102472390954496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105182148 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62123448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43032330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96857726741504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4627685936128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97844705018368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41962157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87981199301280 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104¹ðéÕ2™- _wandbruntime‚2809  _runtime‚2809{MbšÕ /loss/FBt3/detection_loss‚0.8140835287765815 epoch‚1 /loss/FRt3/assignment_loss‚1.336931401285632 2loss/SRqqt3/assignment_loss‚1.0639498165437897 2loss/SRqqt1/assignment_loss‚1.6010007828187485 1loss/FBt2/assignment_loss‚0.34320966515460893 _runtime‚2809.544053722 .loss/FBt1/assignment_loss‚0.39662876816706 1loss/FBt4/assignment_loss‚0.22754322668701768 /loss/FRt4/detection_loss‚2.6115265120478237 trainer/global_step‚2299 0loss/SRqqt3/detection_loss‚1.224883707475315 &loss/total_loss‚43.786119315044246 $ _timestamp‚1.788531825390026e+09 0loss/FRt1/assignment_loss‚3.0112991829713187 0loss/FRt2/assignment_loss‚3.7914160913036716 4loss/FBt3/assignment_loss‚1.3754915838804804e-08 1loss/SRqqt1/detection_loss‚1.9293195204092906 /loss/FBt2/detection_loss‚2.2757959336042406 .lo5¿ùoss/FBt4/detection_loss‚1.124374040402472 0loss/FRt4/assignment_loss‚3.7449868934995987 .loss/FRt1/detection_loss‚2.740486294031143 /loss/FRt2/detection_loss‚2.9217700842888124 0loss/SRqqt2/detection_loss‚2.145455722455625 _step‚186 2loss/SRqqt2/assignment_loss‚2.0097909820852458 -loss/FRt3/detection_loss‚1.62018587688605 2loss/SRqqt4/assignment_loss‚1.5973771686355274 1loss/SRqqt4/detection_loss‚3.2139484915468426 .loss/FBt1/detection_loss‚2.040155610212913rð`›Ú 0loss/FRt4/assignment_loss‚3.7449868934995987 .loss/FRt1/detection_loss‚2.740486294031143 /loss/FRt2/detection_loss‚2.9217700842888124 0loss/SRqqt2/detection_loss‚2.145455722455625 _step‚186 2loss/SRqqt2/assignment_loss‚2.0097909820852458 -loss/FRt3/detection_loss‚1.62018587688605 2loss/SRqqt4/assignment_loss‚1.5973771686355274 1loss/SRqqt4/detection_loss‚3.2139484915468426 .loss/FBt1/detection_loss‚2.040155610212913 /loss/FBt3/detection_loss‚0.8140835287765815 epoch‚1 /loss/FRt3/assignment_loss‚1.336931401285632 2loss/SRqqt3/assignment_loss‚1.0639498165437897 2loss/SRqqt1/assignment_loss‚1.6010007828187485 1loss/FBt2/assignment_loss‚0.34320966515460893 _runtime‚2809.544053722 .loss/FBt1/assignment_loss‚0.39662876816706 1loss/FBt4/assignment_loss‚0.22754322668701768 /loss/FRt4/detection_loss‚2.6115265120478237 trainer/global_step‚2299 0loss/SRqqt3/detection_loss‚1.224883707475315 &loss/total_loss‚43.786119315044246 $ _timestamp‚1.788531825390026e+09 0loss/FRt1/assignment_loss‚3.0112991829713187 0loss/FRt2/assignment_loss‚3.7914160913036716 4loss/FBt3/assignment_loss‚1.3754915838804804e-08 1loss/SRqqt1/detection_loss‚1.9293195204092906 /loss/FBt2/detection_loss‚2.2757959336042406 .loss/FBt4/detection_loss‚1.124374040402472º]Õ§—œ‚(:… û¨ëÔ€†• gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚98.092' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.6184 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿD‰4z‚(:ï û¨ëÔ•¶š disk.sda3.out‚ 90.55078125( disk.nvme0c0n1.out‚54738.11962890625 disk.sda1.in‚0( proc.memory.availableMB‚ 1581.6328125 network.sent‚ 4.604093e+06 network.recv‚940447 disk.sda2.out‚0 disk.dm-0.in‚ 6448.38671875$ memory_percent‚90.34647941589355& disk.nvme0n1.out‚54738.11962890625# disk.nvme0c0n1.in‚ 6448.38671875 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3020.0703125* proc.memory.percent‚18.433046340942383 cpu‚25.240181759167275 proc.cpu.threads‚17) disk./.usagePercent‚79.00771730615847% disk./.usageGB‚1177.2720832824707 disk.sda2.in‚0# disk.dm-0.out‚54738.11962890625 disk.dm-1.in‚0 disk.dm-1.out‚ 90.55078125 disk.sda1.out‚0 disk.drbd1260.out‚ 0.38671875 disk.sda.in‚0 disk.sda.out‚ 90.55078125! disk.nvme0n1.in‚ 6448.38671875 disk.sda3.in‚0ä(eiž‚(:Þ Š©ëÔÄŽÀ˜ disk.drbd1260.in‚0! disk.nvme0n1.in‚ 6448.38671875) proc.memory.availableMB‚ 1582.12109375 cpu‚25.239571782588797% disk./.usageGB‚1177.2720642089844# disk.nvme0c0n1.out‚ 55042.234375 disk.sda.out‚ 91.05078125 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.out‚ 55042.234375" proc.memory.rssMB‚ 3020.0703125* proc.memory.percent‚18.433046340942383 disk.sda1.in‚0( disk./.usagePercent‚79.0077160261208 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 6448.38671875 disk.dm-0.in‚ 6448.38671875 disk.dm-1.out‚ 91.05078125 disk.drbd1260.out‚0.390625 disk.sda.in‚0 disk.dm-0.out‚ 55042.234375 disk.dm-1.in‚0$ memory_percent‚90.34349918365479 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 4.713026e+06 network.recv‚945579 disk.sda3.out‚ 91.05078125ÿ4~œŸ‚(:‘ Š©ëÔベ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.766' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.753200000000003 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üª Oš ‚(: ™©ëÔ—¹§— gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å!˜l¡‚(:á ™©ëÔŒáœ" proc.memory.rssMB‚ 3020.0703125 network.recv‚950611 disk.sda.out‚ 91.49609375 disk.sda2.in‚0! disk.nvme0n1.in‚ 6448.38671875" disk.nvme0n1.out‚ 55252.2578125# disk.nvme0c0n1.in‚ 6448.38671875 disk.dm-0.out‚ 55252.2578125) disk./.usagePercent‚79.00771218600778% disk./.usageGB‚1177.2720069885254$ disk.nvme0c0n1.out‚ 55252.2578125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.390625# memory_percent‚90.3435468673706 disk.sda.in‚0 disk.sda3.out‚ 91.49609375 disk.sda1.out‚0) proc.memory.availableMB‚ 1582.11328125* proc.memory.percent‚18.433046340942383 cpu‚25.238856151704116 proc.cpu.threads‚17 network.sent‚ 4.717574e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 6448.38671875 disk.dm-1.out‚ 91.49609375²ùÃMœ¢‚(:‘ ¨©ëÔÌã…• gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚140.138' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.027600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oµ0gm£‚(:â ¨©ëÔ𢛘 disk.sda3.out‚91.96875" disk.nvme0c0n1.in‚ 6597.3515625" proc.memory.rssMB‚ 3020.0703125) disk./.usagePercent‚79.00771705015094% disk./.usageGB‚1177.2720794677734 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.39453125& disk.nvme0n1.out‚55536.97998046875 network.sent‚ 4.724304e+06 network.recv‚953956 disk.sda.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 6597.3515625( disk.nvme0c0n1.out‚55536.97998046875# memory_percent‚90.3435230255127* proc.memory.percent‚18.433046340942383# disk.dm-0.out‚55536.97998046875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 1582.1171875 cpu‚25.238305869624877 proc.cpu.threads‚17 disk.dm-0.in‚ 6597.3515625 disk.dm-1.out‚91.96875 disk.sda.out‚91.96875$Öùk2¤- _wandbruntime‚2868  _runtime‚2868«4¢Æ¥À & lr-AdamW/pg1‚0.0007946932148612758 & lr-AdamW/pg2‚0.0007946932148612758 trainer/global_step‚2349 $ _timestamp‚1.788531883968447e+09 _runtime‚2868.122691657 _step‚187Wv&˦Š& lr-AdamW/pg1‚0.0007946932148612758 & lr-AdamW/pg2‚0.0007946932148612758 trainer/global_step‚2349 $ _timestamp‚1.788531883968447e+09 _runtime‚2868.122691657 _step‚187»P¡`!2§- _wandbruntime‚2868  _runtime‚2868ì/ƒ×Í'¨ÇO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63461050 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63434708 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚94234076217344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4644936020480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43927805 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63461050 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2868.126249071 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99887219550720 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94451970332365 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63434708 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16703348 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104612717083648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107362513 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98879245905408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42832732 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104613023255552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4725779764736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94452216352801 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43927807 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98878994234880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43927805 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43927807 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4645169688064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282231808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89820203854416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 _step‚188 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4645169688064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42833420 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 $ _timestamp‚1.788531883971534e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94234058214400 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89820186815056 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98878994234880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94234076217344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94234058214400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4725497532928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0®Ù¸2Ò'©ÌO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43927805 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43927807 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4645169688064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282231808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89820203854416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 _step‚188 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4645169688064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42833420 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 $ _timestamp‚1.788531883971534e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94234058214400 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89820186815056 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98878994234880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94234076217344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94234058214400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4725497532928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63461050 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63434708 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚94234076217344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4644936020480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43927805 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63461050 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2868.126249071 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99887219550720 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94451970332365 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63434708 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16703348 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104612717083648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107362513 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98879245905408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42832732 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104613023255552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4725779764736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94452216352801 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43927807 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98878994234880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496¼^LM2ª- _wandbruntime‚2869  _runtime‚2869yÿwE«ÒM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98920244297728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63461400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99928903520768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107433850 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42850243 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633742006295 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94273362254336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89857734211840 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43946080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107407480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63487768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚189 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107407480 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4647133727232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98920495´D“981568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4727781504512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16710057 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4727486703104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63461400 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98920495981568 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94491459178775 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4633970987579 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885318852944193e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63487768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚94273344251392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4647133727232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43946082 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26140188 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2869.448543916 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94273344251392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107433850 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4646900046336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26140186 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94491705199419 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43946082 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104656708965376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0AŸ#ZÝ&¬×M % _timestamp‚1.7885318852944193e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63487768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚94273344251392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4647133727232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43946082 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26140188 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2869.448543916 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94273344251392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107433850 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4646900046336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26140186 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94491705199419 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43946082 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104656708965376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98920244297728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63461400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99928903520768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107433850 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42850243 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633742006295 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94273362254336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89857734211840 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43946080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107407480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63487768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚189 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107407480 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4647133727232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98920495981568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4727781504512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16710057 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4727486703104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63461400 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98920495981568 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94491459178775 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4633970987579 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309½ùŸ&W2­- _wandbruntime‚2869  _runtime‚2869¬;ÏÖ\®Ö 1loss/FBt2/assignment_loss‚0.20864008871089587 /loss/FRt2/detection_loss‚3.3495250696485694 0loss/SRqqt1/detection_loss‚1.918203498041907 /loss/FBt3/detection_loss‚0.5670311051205947 _runtime‚2869.450307309 2loss/SRqqt1/assignment_loss‚1.3052456148488576 1loss/SRqqt4/assignment_loss‚2.498317474828047 0loss/FBt1/assignment_loss‚0.2528828909060264 0loss/SRqqt2/detection_loss‚2.631153312222711 epoch‚1 % _timestamp‚1.7885318852969856e+09 2loss/SRqqt3/assignment_loss‚1.2151726509198115 1loss/SRqqt4/detection_loss‚3.1522068837109734 /loss/FRt2/assignment_loss‚3.627056959000501 3loss/FBt3/assignment_loss‚3.209479560749829e-08 /loss/FRt1/detection_loss‚2.3656388725105084 /loss/FRt4/detection_loss‚3.3639405046900115 %loss/total_loss‚47.38637102413868 trainer/global_step‚2349 0loss/FRt1/assignment_loss‚3.8237331760557076 1loss/FBt4/assignment_loss‚0.10672982055281735 .loss/FBt1/detection_loss‚2.646643957921437 /loss/FRt3/detection_loss‚1.5916386634895676 /loss/FRt3/assignment_loss‚1.658453768021182 1loss/SRqqt3/detection_loss‚0.9047694392194433 .loss/FBt2/detection_loss‚2.876973908681136 /loss/FBt4/detection_loss‚1.5907894276283883 _step‚190 /loss/FRt4/assignment_loss‚4.104338829716046 2loss/SRqqt2/assignment_loss‚1.62728507559874961­Òêa¯Û /loss/FRt2/assignment_loss‚3.627056959000501 3loss/FBt3/assignment_loss‚3.209479560749829e-08 /loss/FRt1/detection_loss‚2.3656388725105084 /loss/FRt4/detection_loss‚3.3639405046900115 %loss/total_loss‚47.38637102413868 trainer/global_step‚2349 0loss/FRt1/assignment_loss‚3.8237331760557076 1loss/FBt4/assignment_loss‚0.10672982055281735 .loss/FBt1/detection_loss‚2.646643957921437 /loss/FRt3/detection_loss‚1.5916386634895676 /loss/FRt3/assignment_loss‚1.658453768021182 1loss/SRqqt3/detection_loss‚0.9047694392194433 .loss/FBt2/detection_loss‚2.876973908681136 /loss/FBt4/detection_loss‚1.5907894276283883 _step‚190 /loss/FRt4/assignment_loss‚4.104338829716046 2loss/SRqqt2/assignment_loss‚1.6272850755987496 1loss/FBt2/assignment_loss‚0.20864008871089587 /loss/FRt2/detection_loss‚3.3495250696485694 0loss/SRqqt1/detection_loss‚1.918203498041907 /loss/FBt3/detection_loss‚0.5670311051205947 _runtime‚2869.450307309 2loss/SRqqt1/assignment_loss‚1.3052456148488576 1loss/SRqqt4/assignment_loss‚2.498317474828047 0loss/FBt1/assignment_loss‚0.2528828909060264 0loss/SRqqt2/detection_loss‚2.631153312222711 epoch‚1 % _timestamp‚1.7885318852969856e+09 2loss/SRqqt3/assignment_loss‚1.2151726509198115 1loss/SRqqt4/detection_loss‚3.1522068837109734¾L¬‰°‚(:… ·©ëÔ𘙠gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚96.211' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)!Lw±‚(:ì ·©ëÔžõÙ disk.drbd1260.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 6597.3515625 disk.dm-0.in‚ 6597.3515625 disk.sda1.out‚0" proc.memory.rssMB‚ 3020.0703125* proc.memory.percent‚18.433046340942383 cpu‚25.237582330728742 network.recv‚958731) disk./.usagePercent‚79.00611393097152 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.39453125 disk.sda.out‚ 92.48046875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 92.48046875$ disk./.usageGB‚1177.248191833496# disk.dm-0.out‚55750.28857421875 disk.sda.in‚0 disk.nvme0n1.in‚ 6597.3515625& disk.nvme0n1.out‚55750.28857421875 disk.sda3.in‚0$ memory_percent‚90.34678936004639) proc.memory.availableMB‚ 1581.58203125 network.sent‚ 4.728654e+06( disk.nvme0c0n1.out‚55750.28857421875 disk.dm-1.out‚ 92.48046875 disk.sda1.in‚0—|Ç{‘²‚(:† Æ©ëÔÒÃŽ• gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚130.548' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.1096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P’˜?h³‚(:Ý Æ©ëÔ®¸˜ disk.sda.out‚ 93.234375 network.sent‚ 4.835519e+06 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚18.433046340942383 network.recv‚964148) disk./.usagePercent‚79.00768760928449 disk.dm-0.in‚ 6597.3515625! disk.dm-0.out‚56247.478515625 disk.sda3.in‚0 disk.sda3.out‚ 93.234375# memory_percent‚90.3437852859497) proc.memory.availableMB‚ 1582.07421875 disk.sda1.in‚0 disk.drbd1260.out‚ 0.3984375 disk.sda2.in‚0 disk.nvme0n1.in‚ 6597.3515625$ disk.nvme0n1.out‚56247.478515625" proc.memory.rssMB‚ 3020.0703125 cpu‚25.23705702912437 proc.cpu.threads‚17 disk.dm-1.out‚ 93.234375 disk.drbd1260.in‚0$ disk./.usageGB‚1177.271640777588" disk.nvme0c0n1.in‚ 6597.3515625& disk.nvme0c0n1.out‚56247.478515625 disk.dm-1.in‚0 disk.sda.in‚0Oµ¨ÿœ´‚(:‘ Õ©ëÔìœå— gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®P ìiµ‚(:Þ Õ©ëԟ컜 disk.sda.out‚ 93.7734375$ disk.nvme0n1.out‚56551.333984375 disk.sda3.out‚ 93.7734375 disk.dm-0.in‚ 6597.3515625 disk.dm-1.in‚0 disk.sda1.in‚0( disk./.usagePercent‚79.0076891453297 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚90.34380912780762( proc.memory.availableMB‚ 1582.0703125" proc.memory.rssMB‚ 3020.0703125 proc.cpu.threads‚17 network.recv‚969244 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 93.7734375 disk.sda1.out‚0* proc.memory.percent‚18.433046340942383 cpu‚25.236372865395 disk.drbd1260.out‚ 0.3984375 disk.nvme0n1.in‚ 6597.3515625" disk.nvme0c0n1.in‚ 6597.3515625& disk.nvme0c0n1.out‚56551.333984375! disk.dm-0.out‚56551.333984375 network.sent‚ 4.841902e+06% disk./.usageGB‚1177.2716636657715 ²ƒ ¶‚(:„ ä©ëÔ˜ëò— gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4662 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­O%9{·‚(:ð ä©ëÔ㊈™ disk.sda3.out‚ 94.17578125 disk.sda1.out‚0 disk.sda.in‚0$ memory_percent‚90.34230709075928" proc.memory.rssMB‚ 3020.0703125 cpu‚25.235657646410537 proc.cpu.threads‚17 network.sent‚ 4.848659e+06% disk./.usageGB‚1177.2717475891113& disk.nvme0n1.out‚56981.95361328125 disk.drbd1260.out‚ 0.40234375* proc.memory.percent‚18.433046340942383) disk./.usagePercent‚79.00769477749546 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 6742.84765625( disk.nvme0c0n1.out‚56981.95361328125 disk.dm-1.in‚0 disk.dm-1.out‚ 94.17578125 disk.sda1.in‚0 disk.drbd1260.in‚0 network.recv‚972409! disk.nvme0n1.in‚ 6742.84765625 disk.dm-0.in‚ 6742.84765625# disk.dm-0.out‚56981.95361328125 disk.sda.out‚ 94.17578125 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1582.316406250s Ž2¸- _wandbruntime‚2927  _runtime‚2927áߨðƹÀ & lr-AdamW/pg1‚0.0007927477684731685 & lr-AdamW/pg2‚0.0007927477684731685 trainer/global_step‚2399 $ _timestamp‚1.788531943276301e+09 _runtime‚2927.429770544 _step‚191Éí%H˺Š& lr-AdamW/pg1‚0.0007927477684731685 & lr-AdamW/pg2‚0.0007927477684731685 trainer/global_step‚2399 $ _timestamp‚1.788531943276301e+09 _runtime‚2927.429770544 _step‚191¿D”Ä2»- _wandbruntime‚2927  _runtime‚2927ý¥ ™#¼ÎO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4743385394688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96198378066944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4743151727104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100941511791104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17052781 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26669486 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106797096861696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚2927.431023005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44841555 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96198378066944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 % _timestamp‚1.7885319432773101e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100941511791104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101971441993216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚2399 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43722267 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17052119 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4743151727104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106796780204032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96426672264509 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109640559 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96426672264509 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109640559 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96198378066944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4743151727104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100941511791104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17052781 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26669486 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106797096861696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚2927.431023005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44841555 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96198378066944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 % _timestamp‚1.7885319432773101e+09Àhð÷2¾- _wandbruntime‚2928  _runtime‚29284_ÏÝ&¿×M S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚100982761853952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109659182 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26680286 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102013125963264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44859832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102013102023168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4827356705792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4745115752960 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64799352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788531944371565e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109685552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96237646100992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43740310 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43739623 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106840769081856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44859830 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745349433856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44859832 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100983013537792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚193 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _runtime‚2928.526243465 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4745349433856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96465915090483 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109685552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91734252083040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64825720 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100983013537792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26680288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44859830 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96237646100992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17060022 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64825720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64799352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106840458728960 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827643118592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731909028087 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273È2yâ&ÀÜM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4745115752960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17059337 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚96237664103936 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96237646100992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17060022 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64825720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64799352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106840458728960 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827643118592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731909028087 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚100982761853952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109659182 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26680286 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102013125963264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44859832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102013102023168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4827356705792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4745115752960 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64799352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788531944371565e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109685552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96237646100992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43740310 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43739623 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106840769081856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44859830 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745349433856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44859832 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100983013537792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚193 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _runtime‚2928.526243465 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4745349433856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96465915090483 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109685552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91734252083040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64825720 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100983013537792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26680288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44859830 PDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚45942500 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚17459760 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 339145216 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚27447140 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚104251139410432 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32004 trainer/global_step‚2417 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4886839617024 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚65721803 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32004 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚3017.141583687 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 _step‚195 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 111664301 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966899712 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚109138002967552 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4790318469195 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 111685012 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚44906518 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193814016 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4803762421760 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚98418012804608 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 111685012 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚382 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859824 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚65742512 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚98418030807552 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 175284736 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚109137827682816 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚27447142 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820464 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193814016 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175811072 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565024768 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175811072 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20711 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚98452539817835 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288769 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859824 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚65742512 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚98418030807552 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 175284736 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚109137827682816 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚27447142 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820464 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193814016 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175811072 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565024768 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175811072 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20711 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚98452539817835 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288769 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚45942500 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚17459760 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 339145216 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚27447140 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚104251139410432 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32004 trainer/global_step‚2417 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4886839617024 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚65721803 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32004 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚3017.141583687 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 _step‚195 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 111664301 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966899712 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚109138002967552 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4790318469195 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 111685012 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚44906518 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193814016 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4803762421760 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚98418012804608 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 111685012 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚382 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚45957882 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚44924174 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885320344227362e+09 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚45957880 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚103252508574720 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059449 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288769 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565024768 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚65751420 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4790790677007 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193814016 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚98448078478848 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175811072 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚44923790 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚104281501164032 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚98478571675279 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚45957882 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚65751420 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 111709302 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚382 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 111688591 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20709 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚17464968 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859824 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚98478382815455 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚27458822 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17465350 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 111709302 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚45957880 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4887313477120 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2417 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 B ûÏI)ÕÃR ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4804412092928 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚98448096481792 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284736 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚98448096481792 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚109168663296512 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4804412092928 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4804236281856 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175811072 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 _runtime‚ 3018.57706137 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚384 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚104281525104128 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 _step‚196 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚93687780998272 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚103252508574720 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚45957882 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚44924174 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885320344227362e+09 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚45957880 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚103252508574720 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059449 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288769 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565024768 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚65751420 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4790790677007 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193814016 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚98448078478848 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175811072 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚44923790 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚104281501164032 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚98478571675279 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚45957882 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚65751420 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 111709302 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚382 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 111688591 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20709 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚17464968 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859824 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚98478382815455 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚27458822 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17465350 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 111709302 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚45957880 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4887313477120 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2417 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_4_of_4‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.1666666567325592 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38923612236976624 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.007861192338168621 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46402567625045776 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.34551653265953064 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02313612401485443 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1904040426015854 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9849710464477539 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01912432163953781 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6439393758773804 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16081079840660095 *jet/accuracy_2_of_4‚0.3512488901615143 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07058726996183395 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_4_of_6‚1 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9849863648414612 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9730602502822876 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5571776032447815 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5063197612762451 6validation_average_jet_accuracy‚0.4385312497615814 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30012640357017517 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19367285072803497 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1175432875752449 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555556207895279 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35085639357566833 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1558193415403366 particle/accuracy_2_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 particle/accuracy_6_of_7‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2586510181427002 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.45485150814056396 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4984435439109802 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5325111150741577 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31111109256744385 jet/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37951815128326416 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0019841270986944437 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9790182709693909 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013006175868213177 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9936984181404114 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.51130211353302 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36234742403030396 particle/accuracy_2_of_4‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897053241729736 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48772725462913513 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 _step‚197 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40828177332878113 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11551457643508911 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9885427951812744 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3513467311859131 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.49119675159454346 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.024848660454154015 particle/accuracy_3_of_6‚1 particle/accuracy_4_of_7‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4379849135875702 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.44476744532585144 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.48076921701431274 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3544856905937195 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11766502261161804 particle/accuracy_2_of_7‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.146749809384346 jet/accuracy_7_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5336230397224426 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 _runtime‚3036.662035355 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.531438410282135 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343Ú»T d421936 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6410117149353027 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19543060660362244 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3483169376850128 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.025844857096672058 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9973359704017639 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 trainer/global_step‚2417 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14693301916122437 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970704317092896 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1961560696363449 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42566969990730286 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0434027798473835 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2036079615354538 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 particle/accuracy_1_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14462970197200775 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22870370745658875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15685372054576874 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25344550609588623 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.525122344493866 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9772849082946777 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_3_of_7‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.028735633939504623 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5651898980140686 particle/accuracy_5_of_5‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0037735849618911743 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.535750150680542 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.491666704416275 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16558130085468292 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 particle/accuracy_2_of_6‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09266416728496552 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007624546531587839 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5392152070999146 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5100484490394592 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9961619973182678 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_1_of_3‚0.8701156973838806 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9126984477043152 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.47453922033309937 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.8928571343421936 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5237536430358887 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1488056778907776 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 jet/accuracy_5_of_5‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12787145376205444 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43883392214775085 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.49093014001846313 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_2_of_3‚0.23114529252052307 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9791667461395264 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22669188678264618 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6881429553031921 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9118801951408386 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929919242858887 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0011337868636474013 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5503267645835876 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.37733399868011475 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.250523179769516 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9795320630073547 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2318965494632721 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4026143550872803 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0.5 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27491313219070435 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4846491515636444 particle/accuracy_7_of_7‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5286291241645813 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.33888882398605347 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14742767810821533 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5416666865348816 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5390981435775757 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9751729369163513 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9872704148292542 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19262436032295227 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10198308527469635 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5023163557052612 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1487276405096054 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.41838401556015015 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.71875 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2123565822839737 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9955271482467651 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.033055298030376434 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5155627131462097 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4886661469936371 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9712302088737488 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.16948997974395752 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4562893211841583 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1230158805847168 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992941617965698 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.017569687217473984 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3726934492588043 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_2_of_2‚0.10870800167322159 particle/accuracy_2_of_3‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19212524592876434 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5005948543548584 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5092405676841736 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18277573585510254 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 % _timestamp‚1.7885320525058625e+09 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06514537334442139 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5665169954299927 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5777475833892822 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9300000071525574 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16567784547805786 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 +particle/specificity‚0.9441080093383789 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.982857346534729 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21705953776836395 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1504632830619812 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.015681004151701927 particle/accuracy_1_of_5‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2648809552192688 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.033007457852363586 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.613636314868927 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5374999642372131 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.911347508430481 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969708919525146 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.525122344493866 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5115740895271301 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20593151450157166 *jet/accuracy_1_of_5‚0.9442524313926697 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9853390455245972 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14788120985031128 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.49388039112091064 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9782193899154663 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5256410241127014 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5246253609657288 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1510273814201355 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4036512076854706 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.7058823704719543 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5500810146331787 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 jet/accuracy_4_of_7‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11831275373697281 jet/accuracy_6_of_7‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48427870869636536 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9905538558959961 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.131226047873497 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0055555556900799274 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3848324418067932 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4228006601333618 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26929256319999695 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007025463972240686 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +particle/sensitivity‚0.3507363498210907 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2043103575706482 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 particle/accuracy_3_of_5‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 *loss/val_total_loss‚45.210284493510265 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20146198570728302 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 +jet/accuracy_3_of_6‚0.09615384787321091 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4655172526836395 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5090284943580627 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 epoch‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07909110933542252 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3181237280368805 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14473684132099152 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1883620023727417 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.01160714216530323 (particle/accuracy‚0.8514437675476074 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3958333432674408 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3772321343421936 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4444444477558136 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.815820574760437 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.327160507440567 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333015441895 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5180554986000061 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 (particle/f_score‚0.42422977089881897 ,"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.375 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677075386047 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9881324768066406 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_5_of_6‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13340245187282562 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.27703240513801575 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5265553593635559 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9907103776931763 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1263580471277237 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812?’ØÚ¡¦ <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0055555556900799274 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3848324418067932 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4228006601333618 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26929256319999695 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007025463972240686 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +particle/sensitivity‚0.3507363498210907 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2043103575706482 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 particle/accuracy_3_of_5‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 *loss/val_total_loss‚45.210284493510265 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20146198570728302 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 +jet/accuracy_3_of_6‚0.09615384787321091 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4655172526836395 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5090284943580627 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 epoch‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07909110933542252 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3181237280368805 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14473684132099152 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1883620023727417 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.01160714216530323 (particle/accuracy‚0.8514437675476074 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3958333432674408 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3772321343421936 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4444444477558136 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.815820574760437 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.327160507440567 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333015441895 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5180554986000061 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 (particle/f_score‚0.42422977089881897 ,"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.375 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9910677075386047 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9881324768066406 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_5_of_6‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13340245187282562 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.27703240513801575 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5265553593635559 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9907103776931763 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1263580471277237 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2638888955116272 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5294674634933472 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14478819072246552 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010416865348816 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3763241469860077 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26254722476005554 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5043894648551941 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 )jet/accuracy_1_of_2‚0.755056619644165 particle/accuracy_2_of_2‚1 particle/accuracy_6_of_6‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4827254116535187 àü'Pw,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1558193415403366 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.47575151920318604 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06313230097293854 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5514858961105347 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5026682615280151 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9780701994895935 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.45874667167663574 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27828043699264526 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18749505281448364 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5151985883712769 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4059912860393524 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23792070150375366 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02752702124416828 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15924635529518127 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3576628565788269 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_4_of_4‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.1666666567325592 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38923612236976624 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.007861192338168621 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46402567625045776 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.34551653265953064 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02313612401485443 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1904040426015854 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9849710464477539 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01912432163953781 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6439393758773804 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16081079840660095 *jet/accuracy_2_of_4‚0.3512488901615143 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07058726996183395 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_4_of_6‚1 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9849863648414612 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9730602502822876 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5571776032447815 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5063197612762451 6validation_average_jet_accuracy‚0.4385312497615814 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30012640357017517 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19367285072803497 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1175432875752449 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555556207895279 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35085639357566833 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1558193415403366 particle/accuracy_2_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 particle/accuracy_6_of_7‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2586510181427002 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.45485150814056396 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4984435439109802 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5325111150741577 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31111109256744385 jet/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37951815128326416 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0019841270986944437 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9790182709693909 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013006175868213177 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9936984181404114 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.51130211353302 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36234742403030396 particle/accuracy_2_of_4‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897053241729736 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48772725462913513 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 _step‚197 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40828177332878113 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11551457643508911 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9885427951812744 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3513467311859131 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.49119675159454346 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.024848660454154015 particle/accuracy_3_of_6‚1 particle/accuracy_4_of_7‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4379849135875702 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.44476744532585144 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.48076921701431274 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3544856905937195 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11766502261161804 particle/accuracy_2_of_7‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.146749809384346 jet/accuracy_7_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5336230397224426 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 _runtime‚3036.662035355 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.531438410282135 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6410117149353027 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19543060660362244 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3483169376850128 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.025844857096672058 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9973359704017639 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 trainer/global_step‚2417 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14693301916122437 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970704317092896 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1961560696363449 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42566969990730286 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0434027798473835 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2036079615354538 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 particle/accuracy_1_of_6‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14462970197200775 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22870370745658875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15685372054576874 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25344550609588623 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.525122344493866 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9772849082946777 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_3_of_7‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.028735633939504623 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5651898980140686 particle/accuracy_5_of_5‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0037735849618911743 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.535750150680542 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.491666704416275 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16558130085468292 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 particle/accuracy_2_of_6‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09266416728496552 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007624546531587839 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5392152070999146 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5100484490394592 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9961619973182678 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_1_of_3‚0.8701156973838806 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9126984477043152 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.47453922033309937 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.8928571343421936 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5237536430358887 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1488056778907776 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 jet/accuracy_5_of_5‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12787145376205444 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43883392214775085 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.49093014001846313 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_2_of_3‚0.23114529252052307 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9791667461395264 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22669188678264618 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6881429553031921 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9118801951408386 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929919242858887 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0011337868636474013 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5503267645835876 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.37733399868011475 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.250523179769516 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9795320630073547 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2318965494632721 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4026143550872803 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0.5 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27491313219070435 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4846491515636444 particle/accuracy_7_of_7‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5286291241645813 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.33888882398605347 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14742767810821533 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5416666865348816 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5390981435775757 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9751729369163513 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9872704148292542 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19262436032295227 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10198308527469635 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5023163557052612 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1487276405096054 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.41838401556015015 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.71875 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2123565822839737 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9955271482467651 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.033055298030376434 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5155627131462097 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4886661469936371 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9712302088737488 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.16948997974395752 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4562893211841583 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1230158805847168 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992941617965698 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.017569687217473984 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3726934492588043 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_2_of_2‚0.10870800167322159 particle/accuracy_2_of_3‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19212524592876434 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5005948543548584 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5092405676841736 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18277573585510254 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 % _timestamp‚1.7885320525058625e+09 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06514537334442139 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5665169954299927 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5777475833892822 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9300000071525574 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16567784547805786 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 +particle/specificity‚0.9441080093383789 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.982857346534729 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21705953776836395 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1504632830619812 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.015681004151701927 particle/accuracy_1_of_5‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2648809552192688 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.033007457852363586 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.613636314868927 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5374999642372131 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.911347508430481 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969708919525146 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.525122344493866 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5115740895271301 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20593151450157166 *jet/accuracy_1_of_5‚0.9442524313926697 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9853390455245972 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14788120985031128 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.49388039112091064 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9782193899154663 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5256410241127014 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5246253609657288 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1510273814201355 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4036512076854706 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.7058823704719543 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5500810146331787 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 jet/accuracy_4_of_7‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11831275373697281 jet/accuracy_6_of_7‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48427870869636536 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9905538558959961 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.131226047873497ÅlÚ™œÛ‚(:‘ ܪëÔ¥Šç— gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-+ö“tÜ‚(:é ܪëÔÓÆí˜ disk.sda.out‚ 97.90234375 disk.nvme0n1.in‚ 6905.53125$ memory_percent‚91.54143333435059* proc.memory.percent‚18.578743934631348 network.recv‚ 1.007797e+06% disk./.usageGB‚1176.8554077148438' disk.nvme0c0n1.out‚59614.8916015625 disk.sda.in‚0( proc.memory.availableMB‚ 1385.8515625 cpu‚25.355413910473878 network.sent‚ 5.094359e+06 disk.dm-0.in‚ 6905.53125" disk.dm-0.out‚59614.8916015625% disk.nvme0n1.out‚59614.8916015625# proc.memory.rssMB‚ 3043.94140625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 97.90234375) disk./.usagePercent‚78.97975385920094 disk.nvme0c0n1.in‚ 6905.53125 disk.dm-1.out‚ 97.90234375 disk.drbd1260.out‚ 0.41406251¦”eÝ‚(:… ëªëÔÞ궖 gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t7rìíÞ‚(:Û ëªëÔ¨›“›) disk./.usagePercent‚78.97971212997285 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 7053.6875 disk.sda3.in‚0& disk.nvme0c0n1.out‚59967.740234375 network.recv‚ 1.013755e+06% disk./.usageGB‚1176.8547859191895 disk.dm-1.out‚ 98.171875 disk.drbd1260.out‚ 0.41796875 disk.sda.in‚0 disk.sda3.out‚ 98.171875 disk.nvme0c0n1.in‚ 7053.6875 disk.sda1.out‚0 disk.drbd1260.in‚0$ disk.nvme0=QEµyn1.out‚59967.740234375# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 proc.cpu.threads‚17 disk.sda.out‚ 98.171875 network.sent‚ 5.218409e+06 disk.dm-0.in‚ 7053.6875! disk.dm-0.out‚59967.740234375 disk.sda2.in‚0$ memory_percent‚91.54300689697266& proc.memory.availableMB‚ 1385.59375 cpu‚25.35421824804822ÐJKß‚(:… úªëÔ댖 gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.705' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.141 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öîò%kà‚(:à úªëÔÉø…™) disk./.usagePercent‚78.98129809664746 disk.sda.out‚ 98.6171875 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 98.6171875 disk.sda1.out‚0 network.sent‚ 5.223282e+06 network.recv‚ 1.017097e+06# disk./.usageGB‚1176.87841796875 disk.dm-0.in‚ 7053.6875! disk.dm-0.out‚60385.544921875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚60385.544921875 disk.sda3.out‚ 98.6171875 disk.nvme0n1.in‚ 7053.6875 disk.drbd1260.out‚ 0.41796875$ memory_percent‚91.54326915740967) proc.memory.availableMB‚ 1385.55078125# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.353108240891515 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 7053.6875& disk.nvme0c0n1.out‚60385.544921875Y|`©2á- _wandbruntime‚3078  _runtime‚3078q&ûÆâÀ & lr-AdamW/pg1‚0.0007907222049252144 & lr-AdamW/pg2‚0.0007907222049252144 trainer/global_step‚2449 % _timestamp‚1.7885320942786143e+09 _runtime‚ 3078.43218592 _step‚198þÃóËãÅ % _timestamp‚1.7885320942786143e+09 _runtime‚ 3078.43218592 _step‚198 & lr-AdamW/pg1‚0.0007907222049252144 & lr-AdamW/pg2‚0.0007907222049252144 trainer/global_step‚2449ÆÅåËv2ä- _wandbruntime‚3078  _runtime‚3078: sÚ'åÔO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104928288669696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110924011862528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100040906282123 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _runtime‚3078.433334064 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46724371 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113424464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4857425797851 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2449 % _timestamp‚1.7885320942796712e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100056799390720 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113398119 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104928036998656 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100056799390720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4955580890112 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4871237607936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66673748 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46724371 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105968430972416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105968407032320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46724373 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46724373 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66673748 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100056817393664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95183480484272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4871471276032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45693350 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100041152302563 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4871471276032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104928036998656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4857654778931 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4955296561664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45694019 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27945552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17747800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100056817393664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582†»„*ß'æÙO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46724373 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66673748 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100056817393664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95183480484272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4871471276032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45693350 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100041152302563 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4871471276032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104928036998656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4857654778931 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4955296561664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45694019 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27945552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17747800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100056817393664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104928288669696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110924011862528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100040906282123 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _runtime‚3078.433334064 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46724371 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113424464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4857425797851 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2449 % _timestamp‚1.7885320942796712e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100056799390720 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113398119 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104928036998656 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100056799390720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4955580890112 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4871237607936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66673748 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46724371 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105968430972416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105968407032320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46724373 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360Ça™2.2ç- _wandbruntime‚3079  _runtime‚3079`6Üä&èÞM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66726809 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113443086 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66726809 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66700440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100096085427712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4957578075136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4859613268125 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 trainer/global_step‚2449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100096085427712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45711558 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113469457 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104969287061504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4873435315200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46742648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46742646 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95221010841696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46742646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4859384286837 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106010114942464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113469457 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45710919 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110967384762368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4957293760000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100080641149181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17754567 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95221027881056 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 _runtime‚3079.632004492 _step‚200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288r‚c téãM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train¥Ê–cu&_batch_end/inactive_split_bytes.all.current‚ 308255232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100096085427712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45711558 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113469457 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104969287061504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4873435315200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46742648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46742646 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95221010841696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46742646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4859384286837 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106010114942464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113469457 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45710919 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110967384762368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4957293760000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100080641149181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17754567 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95221027881056 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 _runtime‚3079.632004492 _step‚200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66726809 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113443086 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66726809 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66700440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100096085427712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4957578075136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4859613268125 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 trainer/global_step‚2449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369Èãm~P2ê- _wandbruntime‚3079  _runtime‚3079s3{YëÓ /loss/FRt4/assignment_loss‚3.859849696574004 1loss/FBt1/assignment_loss‚0.29477219039094843 0loss/FRt1/assignment_loss‚3.5587695224865064 2loss/SRqqt2/assignment_loss‚2.0364912907006563 2loss/SRqqt3/assignment_loss‚1.1908204006031156 -loss/FBt4/detection_loss‚1.49158567448871 1loss/SRqqt1/assignment_loss‚1.321282781072353 1loss/SRqqt3/detection_loss‚1.0252623123427231 1loss/SRqqt4/detection_loss‚2.5699571122725806 trainer/global_step‚2449 $ _timestamp‚1.788532095480276e+09 _runtime‚3079.633677165 0loss/FBt2/assignment_loss‚0.3138622977671912 /loss/SRqqt2/detection_loss‚2.47061585938489 .loss/FRt1/detection_loss‚2.548293306215389 /loss/FRt3/detection_loss‚1.5988686855294203 .loss/FBt1/detection_loss‚2.221722938512501 %loss/total_loss‚45.35454773211468 epoch‚2 1loss/SRqqt4/assignment_loss‚2.468424040824175 1loss/FBt4/assignment_loss‚0.27570493917960603 /loss/FRt2/detection_loss‚2.8669396264212472 -loss/FRt4/detection_loss‚3.03383214318234 /loss/FBt2/detection_loss‚2.1404406785964967 0loss/FRt3/assignment_loss‚1.8221580684185028 4loss/FBt3/assignment_loss‚0.00010906301509550076 1loss/SRqqt1/detection_loss‚1.5781815365741128 _step‚201 0loss/FRt2/assignment_loss‚3.6785081556865147 /loss/FBt3/detection_loss‚0.9880954118755957”Åe^ìØ 1loss/SRqqt1/detection_loss‚1.5781815365741128 _step‚201 0loss/FRt2/assignment_loss‚3.6785081556865147 /loss/FBt3/detection_loss‚0.9880954118755957 /loss/FRt4/assignment_loss‚3.859849696574004 1loss/FBt1/assignment_loss‚0.29477219039094843 0loss/FRt1/assignment_loss‚3.5587695224865064 2loss/SRqqt2/assignment_loss‚2.0364912907006563 2loss/SRqqt3/assignment_loss‚1.1908204006031156 -loss/FBt4/detection_loss‚1.49158567448871 1loss/SRqqt1/assignment_loss‚1.321282781072353 1loss/SRqqt3/detection_loss‚1.0252623123427231 1loss/SRqqt4/detection_loss‚2.5699571122725806 trainer/global_step‚2449 $ _timestamp‚1.788532095480276e+09 _runtime‚3079.633677165 0loss/FBt2/assignment_loss‚0.3138622977671912 /loss/SRqqt2/detection_loss‚2.47061585938489 .loss/FRt1/detection_loss‚2.548293306215389 /loss/FRt3/detection_loss‚1.5988686855294203 .loss/FBt1/detection_loss‚2.221722938512501 %loss/total_loss‚45.35454773211468 epoch‚2 1loss/SRqqt4/assignment_loss‚2.468424040824175 1loss/FBt4/assignment_loss‚0.27570493917960603 /loss/FRt2/detection_loss‚2.8669396264212472 -loss/FRt4/detection_loss‚3.03383214318234 /loss/FBt2/detection_loss‚2.1404406785964967 0loss/FRt3/assignment_loss‚1.8221580684185028 4loss/FBt3/assignment_loss‚0.00010906301509550076ÉÓpI‘í‚(:† ‰«ëÔ¡¯Ô— gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¤-ßnî‚(:ã ‰«ëÔÆ–Äš disk.sda3.out‚ 99.046875) proc.memory.availableMB‚ 1385.51171875 network.recv‚ 1.022297e+06 disk.dm-1.out‚ 99.046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚0.421875 disk.sda2.out‚0# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 network.sent‚ 5.229585e+06% disk./.usageGB‚1176.8016052246094 disk.nvme0c0n1.in‚ 7053.6875# disk.dm-0.out‚60567.14208984375 disk.dm-1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.97614312893802 disk.dm-0.in‚ 7053.6875 disk.sda3.in‚0$ memory_percent‚91.54350757598877 cpu‚25.351948661222266( disk.nvme0c0n1.out‚60567.14208984375 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 99.046875 disk.nvme0n1.in‚ 7053.6875& disk.nvme0n1.out‚60567.14208984375C‹öÛ‘ï‚(:† ˜«ëÔˆ†Ã— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÇÖtð‚(:é ˜«ëԳʚ( disk.nvme0c0n1.out‚60842.51708984375 disk.sda1.out‚0 disk.drbd1260.out‚0.421875& disk.nvme0n1.out‚60842.51708984375 proc.cpu.threads‚17 network.recv‚ 1.027768e+06) disk./.usagePercent‚78.97609423149898 disk.dm-1.out‚ 99.44921875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 99.44921875# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 network.sent‚ 5.337137e+06% disk./.usageGB‚1176.8008766174316 disk.nvme0c0n1.in‚ 7053.6875# disk.dm-0.out‚60842.51708984375 disk.sda1.in‚0 disk.sda.out‚ 99.44921875 disk.nvme0n1.in‚ 7053.6875 disk.sda3.in‚0 disk.dm-0.in‚ 7053.6875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚91.54050350189209) proc.memory.availableMB‚ 1386.00390625 cpu‚25.350750203689927žnr[‘ñ‚(:† §«ëÔª®À— gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gïgò‚(:Ü §«ëÔ˜½Îš disk.dm-0.out‚ 61071.1640625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 1385.234375# proc.memory.rssMB‚ 3043.94140625 network.recv‚ 1.032397e+06% disk./.usageGB‚1176.9117050170898 disk.dm-1.out‚99.90625 disk.sda1.out‚0 disk.nvme0n1.in‚ 7237.6953125$ memory_percent‚91.54520034790039 proc.cpu.threads‚17$ disk.nvme0c0n1.out‚ 61071.1640625 disk.dm-0.in‚ 7237.6953125 disk.sda1.in‚0 disk.sda.out‚99.90625 disk.sda3.in‚0 disk.sda3.out‚99.90625* proc.memory.percent‚18.578743934631348 cpu‚25.349631492114565 network.sent‚ 5.343394e+06) disk./.usagePercent‚78.98353201839126" disk.nvme0c0n1.in‚ 7237.6953125 disk.drbd1260.out‚ 0.4296875 disk.sda2.out‚0" disk.nvme0n1.out‚ 61071.1640625ðM‡0ó‚(:… ¶«ëÔìׇ• gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.645' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.929 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DN$qô‚(:æ ¶«ëÔ‘¸µ˜" disk.nvme0n1.out‚ 61388.1171875 disk.sda1.out‚0# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.348487281975014 network.sent‚ 5.348112e+06) disk./.usagePercent‚78.98535299998231% disk./.usageGB‚1176.9388389587402 disk.nvme0n1.in‚ 7237.6953125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 7237.6953125 disk.dm-1.out‚ 100.2890625 disk.sda1.in‚0 network.recv‚ 1.035265e+06 disk.sda.out‚ 100.2890625 disk.sda2.in‚0 disk.sda3.out‚ 100.2890625 disk.drbd1260.out‚ 0.4296875# memory_percent‚91.5421724319458 disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 61388.1171875 disk.dm-0.in‚ 7237.6953125 disk.dm-0.out‚ 61388.1171875 disk.dm-1.in‚0 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1385.73046875 proc.cpu.threads‚17 disk.sda.in‚0ãÙS2õ- _wandbruntime‚3138  _runtime‚3138çÒ'žÇöÁ % _timestamp‚1.7885321542086852e+09 _runtime‚3138.362177064 _step‚202 & lr-AdamW/pg1‚0.0007886169463418211 & lr-AdamW/pg2‚0.0007886169463418211 trainer/global_step‚2499 ÖÉ1Ì÷Æ _step‚202 & lr-AdamW/pg1‚0.0007886169463418211 & lr-AdamW/pg2‚0.0007886169463418211 trainer/global_step‚2499 % _timestamp‚1.7885321542086852e+09 _runtime‚3138.362177064Êa-2ø- _wandbruntime‚3138  _runtime‚3138i½¨tÙ'ùÓO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102021101240320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46579487 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4969453314560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18093103 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97059998355472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47638121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5055456291328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚102021119243264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 $ _timestamp‚1.788532154209661e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106990806225920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106990554554880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969686982656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68038043 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18093835 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108052629474816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102015362193831 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4955592819439 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102021101240320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113107769108992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4955363838359 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚734 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97060015394832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106990554554880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68011700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113108085766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚203 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47638123 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68011700 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102015608214271 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28485652 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28485650 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309«üÅWúØO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97060015394832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106990554554880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68011700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113108085766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚203 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47638123 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68011700 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102015608214271 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28485652 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28485650 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.c7I²†‡urrent‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102021101240320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46579487 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4969453314560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18093103 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97059998355472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47638121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5055456291328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚102021119243264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 $ _timestamp‚1.788532154209661e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106990806225920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106990554554880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969686982656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68038043 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18093835 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108052629474816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102015362193831 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4955592819439 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102021101240320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113107769108992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4955363838359 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚734 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280ËÅQæ'2û- _wandbruntime‚3139  _runtime‚3139Ëy/?ä&üÞM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4957322327345 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97097545752256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚107032056302080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68064761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28496454 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5057157100544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4971417340416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18101091 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4957551308633 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47656396 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68038392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4971417340416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46597545 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚102060405280256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚102060387277312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102054851040241 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47656396 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5057443512832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885321553227277e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47656398 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46596885 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97097528712896 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47656398 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115721159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107031804617728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4971651021824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108094313444864 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3139.477812983 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68038392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚102060387277312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18100433 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚113151446605312 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107032056302080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68064761 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚204 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115721159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115694788®¨—dé&ýãM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68064761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28496454 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5057157100544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4971417340416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18101091 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4957551308633 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47656396 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68038392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4971417340416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46597545 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚102060405280256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚102060387277312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102054851040241 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47656396 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5057443512832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885321553227277e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47656398 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46596885 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97097528712896 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47656398 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115721159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107031804617728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4971651021824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108094313444864 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3139.477812983 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68038392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚102060387277312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18100433 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚113151446605312 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107032056302080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68064761 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚204 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115721159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115694788 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4957322327345 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97097545752256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚107032056302080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288ÌYÈMø2þ- _wandbruntime‚3139  _runtime‚3139þØçXÿÒ 1loss/SRqqt4/assignment_loss‚1.830113938699166 1loss/FBt1/assignment_loss‚0.35870522200178434 1loss/SRqqt3/detection_loss‚0.9358198660470191 _runtime‚3139.479407008 1loss/SRqqt1/assignment_loss‚1.655463998110927 /loss/FRt4/detection_loss‚2.8063696787275116 1loss/SRqqt4/detection_loss‚2.7030528436104455 /loss/FBt3/detection_loss‚0.9611130022641384 _step‚205 /loss/FRt2/assignment_loss‚4.361042552524143 0loss/FRt3/assignment_loss‚1.7786570815479055 /loss/FRt4/assignment_loss‚4.070166388462329 1loss/SRqqt3/assignment_loss‚1.262592066930873 &loss/total_loss‚47.159411240852826 % _timestamp‚1.7885321553259852e+09 0loss/FBt2/assignment_loss‚0.4510287422304524 .loss/FRt3/detection_loss‚1.529024487777668 1loss/SRqqt1/detection_loss‚1.9791747441097183 .loss/FBt4/detection_loss‚1.282603525810621 2loss/SRqqt2/assignment_loss‚1.8234329714693807 3loss/FBt3/assignment_loss‚0.0004901340442105803 0loss/SRqqt2/detection_loss‚2.818387297066775 epoch‚2 .loss/FRt1/assignment_loss‚3.87078037960776 1loss/FBt4/assignment_loss‚0.17982736399205573 .loss/FRt1/detection_loss‚2.628035681001071 .loss/FBt2/detection_loss‚2.465871302286784 trainer/global_step‚2499 .loss/FRt2/detection_loss‚2.951457158282951 /loss/FBt1/detection_loss‚2.4562008142471314µßö]€× 1loss/SRqqt3/detection_loss‚0.9358198660470191 _runtime‚3139.479407008 1loss/SRqqt1/assignment_loss‚1.655463998110927 /loss/FRt4/detection_loss‚2.8063696787275116 1loss/SRqqt4/detection_loss‚2.7030528436104455 /loss/FBt3/detection_loss‚0.9611130022641384 _step‚205 /loss/FRt2/assignment_loss‚4.361042552524143 0loss/FRt3/assignment_loss‚1.7786570815479055 /loss/FRt4/assignment_loss‚4.070166388462329 1loss/SRqqt3/assignment_loss‚1.262592066930873 &loss/total_loss‚47.159411240852826 % _timestamp‚1.7885321553259852e+09 0loss/FBt2/assignment_loss‚0.4510287422304524 .loss/FRt3/detection_loss‚1.529024487777668 1loss/SRqqt1/detection_loss‚1.9791747441097183 .loss/FBt4/detection_loss‚1.282603525810621 2loss/SRqqt2/assignment_loss‚1.8234329714693807 3loss/FBt3/assignment_loss‚0.0004901340442105803 0loss/SRqqt2/detection_loss‚2.818387297066775 epoch‚2 .loss/FRt1/assignment_loss‚3.87078037960776 1loss/FBt4/assignment_loss‚0.17982736399205573 .loss/FRt1/detection_loss‚2.628035681001071 .loss/FBt2/detection_loss‚2.465871302286784 trainer/global_step‚2499 .loss/FRt2/detection_loss‚2.951457158282951 /loss/FBt1/detection_loss‚2.4562008142471314 1loss/SRqqt4/assignment_loss‚1.830113938699166 1loss/FBt1/assignment_loss‚0.35870522200178434͵Kk–‘‚(:† Å«ëÔ£— gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5794 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gg4΀‚‚(:õ Å«ëÔŒò“š network.sent‚ 5.367523e+06 disk.sda2.in‚0 disk.sda3.out‚ 101.07421875 disk.dm-0.in‚ 7237.6953125 disk.dm-1.in‚0 disk.sda.out‚ 101.07421875' proc.memory.availableMB‚ 1385.703125 cpu‚25.347367446395552 disk.sda2.out‚0& disk.nvme0n1.out‚61867.76611328125 disk.sda3.in‚0# disk.dm-0.out‚61867.76611328125 disk.sda1.in‚0 disk.drbd1260.out‚ 0.43359375 proc.cpu.threads‚17 network.recv‚ 1.040111e+06) disk./.usagePercent‚78.98377624957892% disk./.usageGB‚1176.9153442382812" disk.nvme0c0n1.in‚ 7237.6953125 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚91.54233932495117# proc.memory.rssMB‚ 3043.94140625 disk.nvme0n1.in‚ 7237.6953125( disk.nvme0c0n1.out‚61867.76611328125 disk.dm-1.out‚ 101.07421875 disk.sda1.out‚0* proc.memory.percent‚18.578743934631348ßh›ƒ‚(: Ô«ëÔþ˜¥— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.846' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.569200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ì„‚(:ö Ô«ëÔô¹·™ disk.sda1.in‚0 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1385.68359375# proc.memory.rssMB‚ 3043.94140625 network.recv‚ 1.045345e+06# disk.dm-0.out‚62172.60205078125 disk.sda.out‚ 101.51953125 disk.sda3.out‚ 101.51953125 cpu‚25.346219881744208 proc.cpu.threads‚17( disk./.usagePercent‚78.9895594597774% disk./.usageGB‚1177.0015182495117( disk.nvme0c0n1.out‚62172.60205078125 disk.dm-1.out‚ 101.51953125 disk.sda1.out‚0 disk.drbd1260.out‚ 0.43359375 disk.sda.in‚0 disk.sda2.in‚0 network.sent‚ 5.461977e+06" disk.nvme0c0n1.in‚ 7237.6953125 disk.sda2.out‚0 disk.nvme0n1.in‚ 7237.6953125& disk.nvme0n1.out‚62172.60205078125 disk.sda3.in‚0$ memory_percent‚91.54245853424072* proc.memory.percent‚18.578743934631348 disk.dm-0.in‚ 7237.6953125 disk.dm-1.in‚0/ŠHœ…‚(:‘ ã«ëÔâÐÅ• gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.689' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.537799999999997 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷CÄo†‚(:ê ã«ëÔ¦½¢™ disk.sda3.in‚0 disk.dm-0.in‚ 7368.14453125 disk.sda.out‚ 101.84375 disk.sda2.out‚0$ memory_percent‚91.54253005981445 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 7368.14453125 disk.dm-1.in‚0 disk.dm-1.out‚ 101.84375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 7368.14453125& disk.nvme0n1.out‚62389.19482421875) disk./.usagePercent‚78.99373724672868 disk.sda3.out‚ 101.84375# disk.dm-0.out‚62389.19482421875' proc.memory.availableMB‚ 1385.671875# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.34515200828675 network.sent‚ 5.468241e+06% disk./.usageGB‚1177.0637702941895( disk.nvme0c0n1.out‚62389.19482421875 disk.sda1.out‚0 disk.drbd1260.out‚0.4375 disk.sda2.in‚0 network.recv‚ 1.0500Á ½Ç64e+06ø€Ã‡‚(:… ò«ëÔ¼¸˜ gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0384 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FºÕ°€ˆ‚(:õ ò«ëÔìÑëš# proc.memory.rssMB‚ 3043.94140625( disk.nvme0c0n1.out‚62606.74951171875 disk.dm-1.out‚ 102.44921875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.4375 disk.sda.in‚0 cpu‚25.343988446585584) disk./.usagePercent‚79.00212482157299% disk./.usageGB‚1177.1887512207031 disk.dm-0.in‚ 7368.14453125 disk.dm-1.in‚0! disk.nvme0n1.in‚ 7368.14453125& disk.nvme0n1.out‚62606.74951171875 disk.sda3.out‚ 102.44921875( proc.memory.availableMB‚ 1385.8984375 network.recv‚ 1.053368e+06 disk.sda1.in‚0 disk.sda.out‚ 102.44921875 disk.sda2.out‚0* proc.memory.percent‚18.578743934631348 proc.cpu.threads‚17 network.sent‚ 5.475005e+06# disk.nvme0c0n1.in‚ 7368.14453125# disk.dm-0.out‚62606.74951171875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚91.54114723205566Ö£±2‰- _wandbruntime‚3197  _runtime‚3197ï¶u\ÆŠÀ trainer/global_step‚2549 $ _timestamp‚1.788532213285324e+09 _runtime‚3197.438936263 _step‚206 & lr-AdamW/pg1‚0.0007864324314557218 & lr-AdamW/pg2‚0.0007864324314557218™ƒâŽË‹Å & lr-AdamW/pg1‚0.0007864324314557218 & lr-AdamW/pg2‚0.0007864324314557218 trainer/global_step‚2549 $ _timestamp‚1.788532213285324e+09 _runtime‚3197.438936263 _step‚206ÎJ:Ã2Œ- _wandbruntime‚3197  _runtime‚3197Þ0Ú'ÔO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29025752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117901523 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 _step‚207 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103989818105539 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103990064125979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5067669021184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109053323782144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5067902689280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115292222883840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5155106383872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5067902689280 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48551873 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5053301878867 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5155394906624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103985421092864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115291910420992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98936533266032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69375995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109053072111104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29025750 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69375995 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103985421092864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103985403089920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48551873 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110136804037120 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103985403089920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69349652 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47465418 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48551871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5067669021184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885322132867382e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109053323782144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117927868 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110136827977216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Îû'Äß'ŽÙO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29025750 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69375995 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103985421092864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103985403089920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48551873 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110136804037120 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103985403089920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69349652 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47465418 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48551871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5067669021184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885322132867382e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109053323782144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117927868 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110136827977216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29025752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117901523 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 _step‚207 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103989818105539 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103990064125979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5067669021184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109053323782144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5067902689280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115292222883840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5155106383872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5067902689280 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48551873 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5053301878867 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5155394906624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103985421092864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115291910420992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98936533266032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69375995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109053072111104 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47483380 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29036552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104029306951949 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98974063623456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104024689126912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29036554 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48570146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110178488007168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109094573858304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69376344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5157098617856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚3198.750092687 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109094573858304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5069633047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18446828 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104024707129856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110178511947264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5069866728448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117972861 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69376344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48570148 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5055260367853 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69402713 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109094322173952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5157393418752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885322145951548e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69402713 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5055489349141 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5069866728448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47484093 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36‹!¸ÒÒ‘ãM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48570146 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110178488007168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109094573858304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69376344 ]KDeviceStatsMonitor.on_ì²Ë%train_batch_end/inactive_split_bytes.small_pool.freed‚ 5157098617856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚3198.750092687 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109094573858304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5069633047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18446828 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104024707129856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110178511947264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5069866728448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117972861 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69376344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48570148 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5055260367853 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69402713 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109094322173952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5157393418752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885322145951548e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69402713 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5055489349141 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5069866728448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47484093 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48570146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115335586625024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104024707129856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47483380 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29036552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104029306951949 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98974063623456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104024689126912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29036554 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ÐqPƒ2’- _wandbruntime‚3198  _runtime‚3198Ñfžš]“× 1loss/SRqqt2/detection_loss‚2.5734972932509015 epoch‚2 1loss/SRqqt4/assignment_loss‚1.837046645085017 0loss/FBt1/assignment_loss‚0.3871907902596528 .loss/FRt4/detection_loss‚3.255309513636998 0loss/FRt2/assignment_loss‚3.7783067280595954 0loss/FBt2/assignment_loss‚0.2603876983126554 -loss/FRt1/detection_loss‚2.31376285971822 1loss/SRqqt1/detection_loss‚1.4663669701133455 /loss/FRt1/assignment_loss‚3.776356455442068 1loss/FBt4/assignment_loss‚0.16964705956543785 /loss/FBt1/detection_loss‚2.9395812153816223 /loss/FRt4/assignment_loss‚4.218462291217985 trainer/global_step‚2549 _runtime‚3198.752774483 /loss/FRt2/detection_loss‚3.2831324203447863 2loss/SRqqt1/assignment_loss‚1.5424037357526166 2loss/SRqqt2/assignment_loss‚1.7947373251829828 3loss/FBt3/assignment_loss‚6.006193194449142e-05 1loss/SRqqt3/detection_loss‚1.1949047393032484 /loss/FBt2/detection_loss‚2.6537431702017784 2loss/SRqqt3/assignment_loss‚1.1218360944620365 1loss/SRqqt4/detection_loss‚3.2381115158398948 /loss/FBt3/detection_loss‚0.8239503455432978 %loss/total_loss‚47.15413392366542 % _timestamp‚1.7885322145987597e+09 _step‚209 0loss/FRt3/assignment_loss‚1.7287894455591837 /loss/FRt3/detection_loss‚1.6347189066807428 /loss/FBt4/detection_loss‚1.1618306428194045ÔTntb”Ü /loss/FRt1/assignment_loss‚3.776356455442068 1loss/FBt4/assignment_loss‚0.16964705956543785 /loss/FBt1/detection_loss‚2.9395812153816223 /loss/FRt4/assignment_loss‚4.218462291217985 trainer/global_step‚2549 _runtime‚3198.752774483 /loss/FRt2/detection_loss‚3.2831324203447863 2loss/SRqqt1/assignment_loss‚1.5424037357526166 2loss/SRqqt2/assignment_loss‚1.7947373251829828 3loss/FBt3/assignment_loss‚6.006193194449142e-05 1loss/SRqqt3/detection_loss‚1.1949047393032484 /loss/FBt2/detection_loss‚2.6537431702017784 2loss/SRqqt3/assignment_loss‚1.1218360944620365 1loss/SRqqt4/detection_loss‚3.2381115158398948 /loss/FBt3/detection_loss‚0.8239503455432978 %loss/total_loss‚47.15413392366542 % _timestamp‚1.7885322145987597e+09 _step‚209 0loss/FRt3/assignment_loss‚1.7287894455591837 /loss/FRt3/detection_loss‚1.6347189066807428 /loss/FBt4/detection_loss‚1.1618306428194045 1loss/SRqqt2/detection_loss‚2.5734972932509015 epoch‚2 1loss/SRqqt4/assignment_loss‚1.837046645085017 0loss/FBt1/assignment_loss‚0.3871907902596528 .loss/FRt4/detection_loss‚3.255309513636998 0loss/FRt2/assignment_loss‚3.7783067280595954 0loss/FBt2/assignment_loss‚0.2603876983126554 -loss/FRt1/detection_loss‚2.31376285971822 1loss/SRqqt1/detection_loss‚1.4663669701133455Ñ;ê`‘•‚(:† ¬ëÔ¾À•— gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.396' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2792 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÚÆX€–‚(:õ ¬ëÔ¸¤Ëš# disk.dm-0.out‚62814.88623046875 disk.dm-1.in‚0# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.3429423072538 network.sent‚ 5.479483e+06 disk.drbd1260.in‚0 disk.sda.out‚ 103.1328125 disk.sda2.out‚0( disk.nvme0c0n1.out‚62814.88623046875 disk.dm-1.out‚ 103.1328125 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.058214e+06 disk.drbd1260.out‚ 0.44140625 disk.sda2.in‚0! disk.nvme0n1.in‚ 7368.14453125& disk.nvme0n1.out‚62814.88623046875 disk.sda3.in‚0 disk.dm-0.in‚ 7368.14453125$ memory_percent‚91.54446125030518) disk./.usagePercent‚79.00055139926755 disk.sda.in‚0) proc.memory.availableMB‚ 1385.35546875 proc.cpu.threads‚17% disk./.usageGB‚1177.1653060913086 disk.sda3.out‚ 103.1328125# disk.nvme0c0n1.in‚ 7368.14453125nê|—‚(:„ ¬ëÔÖÙ¸š gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.455' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.691 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N¼ˆz˜‚(:ï ¬ëÔ‰œóœ disk.dm-1.out‚ 103.6328125" memory_percent‚91.544508934021# proc.memory.rssMB‚ 3043.94140625 network.recv‚ 1.063771e+06) disk./.usagePercent‚79.01753698714933 disk.sda1.out‚0 disk.sda.in‚0 disk.dm-0.in‚ 7368.14453125 disk.dm-1.in‚0 cpu‚25.341863497612664 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 103.6328125! disk.dm-0.out‚63263.271484375* proc.memory.percent‚18.578743934631348 network.sent‚ 5.586843e+06% disk./.usageGB‚1177.4184036254883 disk.drbd1260.out‚ 0.44140625 disk.sda.out‚ 103.6328125! disk.nvme0n1.in‚ 7368.14453125$ disk.nvme0n1.out‚63263.271484375 disk.sda3.in‚0) proc.memory.availableMB‚ 1385.34765625 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 7368.14453125& disk.nvme0c0n1.out‚63263.271484375;l»™‚(:„ Ÿ¬ëÔ÷Ж gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô:)’xš‚(:í Ÿ¬ëÔÈÙÌš% disk./.usageGB‚1177.4417114257812" disk.nvme0c0n1.in‚ 7476.1484375' disk.nvme0c0n1.out‚63697.4013671875 disk.dm-0.in‚ 7476.1484375 disk.drbd1260.out‚ 0.4453125 disk.sda.out‚ 104.04296875 cpu‚25.34086994999292) disk./.usagePercent‚79.01910119318353 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚91.5445327758789* proc.memory.percent‚18.578743934631348 disk.sda2.out‚0 disk.nvme0n1.in‚ 7476.1484375% disk.nvme0n1.out‚63697.4013671875 disk.drbd1260.in‚0 disk.sda.in‚0& proc.memory.availableMB‚ 1385.34375# proc.memory.rssMB‚ 3043.94140625 proc.cpu.threads‚17 network.sent‚ 5.59507e+06 network.recv‚ 1.068414e+06 disk.sda3.out‚ 104.04296875" disk.dm-0.out‚63697.4013671875 disk.dm-1.in‚0 disk.dm-1.out‚ 104.04296875¤¹kþ››‚(: ®¬ëÔ­“Š— gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.846' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.569200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éìËv|œ‚(:ñ ®¬ëÔ¨šÐ™ disk.dm-1.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚64064.9990234375 disk.sda3.in‚0 disk.sda3.out‚ 104.66796875# disk.nvme0c0n1.in‚ 7507.37109375' disk.nvme0c0n1.out‚64064.9990234375# memory_percent‚91.5445327758789$ disk./.usageGB‚1177.441692352295" disk.dm-0.out‚64058.1240234375 disk.dm-1.out‚ 104.66796875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.4453125 disk.sda.out‚ 104.66796875 disk.sda2.in‚0& proc.memory.availableMB‚ 1385.34375 network.sent‚ 5.598093e+06 network.recv‚ 1.071886e+06 disk.dm-0.in‚ 7507.37109375 disk.sda1.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 7507.37109375# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348) disk./.usagePercent‚79.01909991314587 disk.drbd1260.in‚0 cpu‚25.339863180001302 proc.cpu.threads‚17ü˜}t2- _wandbruntime‚3257  _runtime‚3257ìTGÇžÁ _step‚210 & lr-AdamW/pg1‚0.0007841691155165448 & lr-AdamW/pg2‚0.0007841691155165448 trainer/global_step‚2599 % _timestamp‚1.7885322732773993e+09 _runtime‚3257.430857036ξûÌŸÆ trainer/global_step‚2599 % _timestamp‚1.7885322732773993e+09 _runtime‚3257.430857036 _step‚210 & lr-AdamW/pg1‚0.0007841691155165448 & lr-AdamW/pg2‚0.0007841691155165448Ò¸€x2 - _wandbruntime‚3257  _runtime‚3257˜WkºÛ'¡ÕO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5255058375168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105949722942464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70713947 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70687604 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105949722942464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5166118395904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49465621 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5165884727808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105949704939520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29565850 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5151239919375 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120153225 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111115841338368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111115841338368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5166118395904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48352898 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117476373377536 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49465623 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255346897920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105964520037687 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112221026479616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100813051137232 _runtime‚3257.432076068 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70713947 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105964274017247 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48353580 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29565852 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚2599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49465623 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70687604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111115589667328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18787048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117476060914688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5151468900455 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309.uqp¢ÚO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49465621 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5165884727808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105949704939520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29565850 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5151239919375 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120153225 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111115841338368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111115841338368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5166118395904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48352898 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117476373377536 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49465623 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255346897920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105964520037687 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 eQDeviceStatsÀùmpMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112221026479616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100813051137232 _runtime‚3257.432076068 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70713947 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105964274017247 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48353580 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29565852 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚2599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49465623 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70687604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111115589667328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18787048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117476060914688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5151468900455 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5255058375168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105949722942464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70713947 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70687604 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105949722942464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5166118395904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273Ó|9Þ2£- _wandbruntime‚3258  _runtime‚3258Š2Žræ&¤àM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18794286 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120224563 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5153427389649 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29576652 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100850581494656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚3258.383192521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111157091414528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48370938 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112262686509568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106003762863657 trainer/global_step‚2599 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105988990976512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49483896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120198192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49483898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29576654 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117520051157504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5168082435072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120198192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885322742290478e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5167848753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120224563 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105989008979456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70740665 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 _step‚212 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117519751290880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100850564455296Å6ˆÍë&¥åM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70740665 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5167848753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49483898 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48371607 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105989008979456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70740665 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 _step‚212 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117519751290880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100850564455296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18794286 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120224563 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5153427389649 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29576652 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100850581494656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚3258.383192521 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111157091414528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48370938 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112262686509568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106003762863657 trainer/global_step‚2599 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105988990976512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49483896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120198192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49483898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29576654 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117520051157504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5168082435072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120198192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885322742290478e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5167848753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120224563 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113178107223552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5264100434432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19132872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107914006789120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5264334102528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50379373 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5264334102528 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107914024792064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚215 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30105952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119660213614592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107938975949395 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49239514 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5355301095424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5249177959883 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122404927 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50379371 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72025556 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7885323304722295e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113178358894592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113178107223552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50379373 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19133562 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102689551969072 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72025556 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102689569008432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚2649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5355012572672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚114305201041920 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49238822 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49238822 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113178107223552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5264100434432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19132872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107914006789120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5264334102528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50379373 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5264334102528 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107914024792064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚215 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30105952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119660213614592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107938975949395 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49239514 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5355301095424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5249177959883 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122404927 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50379371 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72025556 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7885323304722295e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113178358894592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113178107223552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50379373 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19133562 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102689551969072 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72025556 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102689569008432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚2649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345×!ý´‘µ‚(:† ê¬ëÔÀƒ˜ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾@†}¶‚(:ò ê¬ëԦ㚠disk.sda.in‚0 disk.sda.out‚ 106.515625 disk.sda2.in‚0* proc.memory.percent‚18.578743934631348 network.sent‚ 5.722068e+06% disk./.usageGB‚1177.4187622070312 disk.drbd1260.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 7688.31640625) proc.memory.availableMB‚ 1214.00390625 cpu‚25.335550771632352 proc.cpu.threads‚17) disk./.usagePercent‚79.01756105185754 disk.sda3.in‚0 disk.drbd1260.out‚0.453125& disk.nvme0n1.out‚65039.00732421875$ memory_percent‚92.59030818939209# proc.memory.rssMB‚ 3043.94140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 106.515625# disk.nvme0c0n1.in‚ 7688.31640625( disk.nvme0c0n1.out‚65039.00732421875 disk.dm-0.in‚ 7688.31640625 disk.dm-1.in‚0 disk.dm-1.out‚ 106.515625 network.recv‚ 1.088995e+06# disk.dm-0.out‚65039.00732421875ÖËê¾2·- _wandbruntime‚3315  _runtime‚3315 Âtlæ&¸àM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122476265 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5266298141696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30116752 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72078617 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114346908952064 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119704204135936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102727099365856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5266064460288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5251136448869 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50397646 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107978464796013 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5266298141696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113219357286400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7885323314839888e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113219608970752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50397648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72052248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30116754 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19140246 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107953292826112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119703895880704 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102727082326496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 trainer/global_step‚2649 _runtime‚3315.638492866 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72052248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122449894 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107953310829056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5357295183872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113219608970752 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚709 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49257000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49256289 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114346885011968 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107953310829056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107978218775365 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122476265 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50397646 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5266064460288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5251365430157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122449894 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5357010868736rø{å¹åM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122449894 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5357010868736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122476265 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5266298141696 REDeviceStatkÚ1$sMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30116752 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72078617 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114346908952064 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119704204135936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102727099365856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5266064460288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5251136448869 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50397646 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107978464796013 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5266298141696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113219357286400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7885323314839888e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113219608970752 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50397648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72052248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30116754 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19140246 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107953292826112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119703895880704 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102727082326496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 trainer/global_step‚2649 _runtime‚3315.638492866 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72052248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122449894 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107953310829056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5357295183872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113219608970752 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚709 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49257000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49256289 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114346885011968 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107953310829056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107978218775365 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122476265 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50397646 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5266064460288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5251365430157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104ØT?¦À2º- _wandbruntime‚3315  _runtime‚3315’ô:÷]»× 2loss/SRqqt1/assignment_loss‚1.4598803952060364 0loss/SRqqt3/detection_loss‚0.933114831491063 /loss/FBt1/detection_loss‚2.3749605752527714 2loss/SRqqt4/assignment_loss‚2.3727969601750374 1loss/FBt4/assignment_loss‚0.15069790616346582 % _timestamp‚1.7885323314868345e+09 _runtime‚3315.640226115 0loss/FBt1/assignment_loss‚0.3332687813576456 /loss/FRt3/detection_loss‚1.5273449628413478 .loss/FRt4/detection_loss‚2.955122137511218 1loss/SRqqt1/detection_loss‚1.7287938920436081 _step‚217 /loss/FRt1/assignment_loss‚4.013195839524269 0loss/FRt2/assignment_loss‚3.5830188451274747 .loss/FRt4/assignment_loss‚3.68967072168986 /loss/FBt2/detection_loss‚2.5681101977825165 /loss/FBt4/detection_loss‚1.6533715912216418 epoch‚2 1loss/FBt2/assignment_loss‚0.09303761504634167 1loss/SRqqt4/detection_loss‚2.3824948271115622 trainer/global_step‚2649 0loss/FRt3/assignment_loss‚1.4991488600079017 2loss/FBt3/assignment_loss‚0.010418843763199442 /loss/FRt1/detection_loss‚2.5451445430517197 1loss/SRqqt2/detection_loss‚2.6697157908368996 /loss/FBt3/detection_loss‚1.1392762284529836 .loss/FRt2/detection_loss‚2.976948884225661 2loss/SRqqt2/assignment_loss‚2.0125886814461813 2loss/SRqqt3/assignment_loss‚1.1475158729202424 %loss/total_loss‚45.81963778425065!5ÿÄb¼Ü 0loss/FRt3/assignment_loss‚1.4991488600079017 2loss/FBt3/assignment_loss‚0.010418843763199442 /loss/FRt1/detection_loss‚2.5451445430517197 1loss/SRqqt2/detection_loss‚2.6697157908368996 /loss/FBt3/detection_loss‚1.1392762284529836 .loss/FRt2/detection_loss‚2.976948884225661 2loss/SRqqt2/assignment_loss‚2.0125886814461813 2loss/SRqqt3/assignment_loss‚1.1475158729202424 %loss/total_loss‚45.81963778425065 2loss/SRqqt1/assignment_loss‚1.4598803952060364 0loss/SRqqt3/detection_loss‚0.933114831491063 /loss/FBt1/detection_loss‚2.3749605752527714 2loss/SRqqt4/assignment_loss‚2.3727969601750374 1loss/FBt4/assignment_loss‚0.15069790616346582 % _timestamp‚1.7885323314868345e+09 _runtime‚3315.640226115 0loss/FBt1/assignment_loss‚0.3332687813576456 /loss/FRt3/detection_loss‚1.5273449628413478 .loss/FRt4/detection_loss‚2.955122137511218 1loss/SRqqt1/detection_loss‚1.7287938920436081 _step‚217 /loss/FRt1/assignment_loss‚4.013195839524269 0loss/FRt2/assignment_loss‚3.5830188451274747 .loss/FRt4/assignment_loss‚3.68967072168986 /loss/FBt2/detection_loss‚2.5681101977825165 /loss/FBt4/detection_loss‚1.6533715912216418 epoch‚2 1loss/FBt2/assignment_loss‚0.09303761504634167 1loss/SRqqt4/detection_loss‚2.3824948271115622 trainer/global_step‚2649Ù(¿Ô½‚(:… ù¬ëÔ•àΗ gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6596 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼Sµ~¾‚(:ó ù¬ëÔô‡Ô˜% disk./.usageGB‚1177.4187469482422 disk.sda.out‚ 106.98046875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 106.98046875( disk./.usagePercent‚79.0175600278274# disk.nvme0c0n1.in‚ 7688.31640625! disk.dm-0.out‚65368.400390625 disk.dm-1.in‚0 disk.dm-1.out‚ 106.98046875$ disk.nvme0n1.out‚65368.400390625$ memory_percent‚92.58763790130615) proc.memory.availableMB‚ 1214.44140625 network.sent‚ 5.819527e+06& disk.nvme0c0n1.out‚65368.400390625 disk.dm-0.in‚ 7688.31640625 disk.sda1.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.45703125! disk.nvme0n1.in‚ 7688.31640625 disk.sda3.in‚0 cpu‚25.334385677225942 network.recv‚ 1.095455e+06§4P‘¿‚(:† ˆ­ëÔœ“Ó– gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.998' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9996 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô,©|À‚(:ñ ˆ­ëÔ»‡¤š# disk./.usageGB‚1177.44242477417 disk.sda1.out‚0 disk.drbd1260.out‚ 0.45703125 disk.sda.out‚ 107.56640625 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 disk.sda1.in‚0 disk.drbd1260.in‚0$ disk.nvme0n1.out‚65737.955078125 disk.sda3.in‚0 disk.sda3.out‚ 107.56640625! disk.dm-0.out‚65737.955078125) proc.memory.availableMB‚ 1214.19140625 cpu‚25.333394637856014 network.recv‚ 1.100502e+06 disk.sda.in‚0& disk.nvme0c0n1.out‚65737.955078125 disk.dm-0.in‚ 7688.31640625 disk.dm-1.in‚0 disk.dm-1.out‚ 107.56640625 network.sent‚ 5.916411e+06) disk./.usagePercent‚79.01914906659243! disk.nvme0n1.in‚ 7688.31640625# disk.nvme0c0n1.in‚ 7688.31640625# memory_percent‚92.5891637802124 proc.cpu.threads‚17 Jy'‘Á‚(:† —­ëÔô¡“˜ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.697' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7394 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÅNv~‚(:ó —­ëÔæþœ# memory_percent‚92.5922155380249 network.recv‚ 1.105178e+06 disk.sda2.out‚0( disk.nvme0c0n1.out‚66074.73291015625# disk.dm-0.out‚66074.73291015625 disk.drbd1260.in‚0% disk./.usageGB‚1177.4424438476562 disk.sda2.in‚0! disk.nvme0n1.in‚ 7688.31640625 disk.sda3.out‚ 107.9765625# disk.nvme0c0n1.in‚ 7688.31640625 disk.dm-1.in‚0 cpu‚25.33239587963866 proc.cpu.threads‚17 network.sent‚ 5.92264e+06) disk./.usagePercent‚79.01915034663011 disk.drbd1260.out‚ 0.4609375 disk.sda.in‚0 disk.sda.out‚ 107.9765625) proc.memory.availableMB‚ 1213.69140625# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348& disk.nvme0n1.out‚66074.73291015625 disk.sda3.in‚0 disk.dm-0.in‚ 7688.31640625 disk.dm-1.out‚ 107.9765625 disk.sda1.in‚0 disk.sda1.out‚08ÍÃ!2Ã- _wandbruntime‚3374  _runtime‚3374Kˆw—ÇÄÁ _runtime‚3374.253955875 _step‚218 & lr-AdamW/pg1‚0.0007794079834892801 & lr-AdamW/pg2‚0.0007794079834892801 trainer/global_step‚2699 % _timestamp‚1.7885323901003933e+09šÌ6ñÌÅÆ trainer/global_step‚2699 % _timestamp‚1.7885323901003933e+09 _runtime‚3374.253955875 _step‚218 & lr-AdamW/pg1‚0.0007794079834892801 & lr-AdamW/pg2‚0.0007794079834892801Ú¤Thþ2Æ- _wandbruntime‚3374  _runtime‚3374#éƒÜ'ÇÖO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109913185840663 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121844381105152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121844693568000 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73363508 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5455270083584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51293121 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30646052 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104566086879632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104566069840272 % _timestamp‚1.7885323901015816e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116389399544320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109878326641664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5362549809152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50123401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30646050 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362549809152 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116389423484416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5347344981471 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73389851 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73363508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124656629 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5454981560832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5347116000391 trainer/global_step‚2699 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109878326641664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109878308638720 _step‚219 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51293123 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5362316141056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115240624779776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51293123 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚109913431861103 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109878308638720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115240876450816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19476635 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115240624779776 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 _runtime‚3374.255302094 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51293121 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115240624779776 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 _runtime‚3374.255302094 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51293121 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121844693568000 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73363508 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5455270083584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51293121 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30646052 OIDeviceStatsMonitor.on_train_batc7 Ó²Eh_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104566086879632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104566069840272 % _timestamp‚1.7885323901015816e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116389399544320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109878326641664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5362549809152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50123401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30646050 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362549809152 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116389423484416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5347344981471 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73389851 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73363508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124656629 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5454981560832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5347116000391 trainer/global_step‚2699 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109878326641664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104ÛufÊ‘É‚(:† ¦­ëÔ—Ô¦ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.517' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7034 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öŒ¿¬xÊ‚(:í ¦­ëÔ¹æÏ¡ network.recv‚ 1.108397e+06# disk.dm-0.out‚66316.21337890625 disk.sda3.out‚ 108.3046875# memory_percent‚92.5894021987915) proc.memory.availableMB‚ 1214.15234375* proc.memory.percent‚18.578743934631348 cpu‚25.331376385922553 proc.cpu.threads‚17 disk.dm-1.out‚ 108.3046875 disk.sda2.in‚0 disk.nvme0n1.in‚ 7877.328125& disk.nvme0n1.out‚66316.21337890625 disk.sda3.in‚0# proc.memory.rssMB‚ 3043.94140625 network.sent‚ 5.927508e+06) disk./.usagePercent‚79.01914906659243# disk./.usageGB‚1177.44242477417 disk.dm-0.in‚ 7877.328125 disk.sda1.in‚0 disk.drbd1260.out‚ 0.4609375 disk.sda.out‚ 108.3046875! disk.nvme0c0n1.in‚ 7877.328125( disk.nvme0c0n1.out‚66316.21337890625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0ö«D’2Ë- _wandbruntime‚3375  _runtime‚3375Þ¤?Ææ&ÌàM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51311396 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109917612678656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5349074489377 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124727967 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109917594675712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115281874842624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124727967 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51311398 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50140187 trainer/global_step‚2699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73390200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109917612678656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885323910825179e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109917594675712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73390200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19484009 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51311398 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116431107454464 _runtime‚3375.236765063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5364513848320 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121888367587840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124701596 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124701596 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73416569 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5457260133376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73416569 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚220 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5364280166912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116431083514368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104603600197696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5456973721088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19483335 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121888057235456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115282126526976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5364513848320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30656854 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5349303470665 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5457260133376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73416569 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚220 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5364280166912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116431083514368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104603600197696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5456973721088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19483335 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121888057235456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115282126526976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5364513848320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30656854 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5349303470665 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5349074489377 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124727967 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109917594675712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115281874842624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124727967 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51311398 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50140187 trainer/global_step‚2699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73390200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109917612678656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885323910825179e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109917594675712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73390200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19484009 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51311398 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116431107454464 _runtime‚3375.236765063Üj2ïM2Î- _wandbruntime‚3375  _runtime‚3375¹G[uXÏÒ trainer/global_step‚2699 2loss/SRqqt4/assignment_loss‚1.7448205169704225 /loss/FRt2/detection_loss‚3.0564541128965526 /loss/FRt2/assignment_loss‚3.664677280646104 0loss/FBt1/assignment_loss‚0.2253778815660515 2loss/FBt3/assignment_loss‚0.010990529208562805 %loss/total_loss‚46.90986685889342 /loss/FRt3/assignment_loss‚1.743161570887233 /loss/FRt4/detection_loss‚2.9508929657936096 1loss/SRqqt1/detection_loss‚1.8986784979408862 2loss/SRqqt2/assignment_loss‚1.8840268990572762 0loss/FBt2/assignment_loss‚0.3388580597126065 /loss/FRt1/detection_loss‚2.6428627127950843 /loss/FBt1/detection_loss‚2.8984143882989883 _runtime‚3375.238352116 epoch‚2 % _timestamp‚1.7885323910849938e+09 0loss/FRt1/assignment_loss‚3.3957373824986545 /loss/FRt4/assignment_loss‚3.683554995059967 1loss/SRqqt3/assignment_loss‚1.065829457334158 .loss/FRt3/detection_loss‚1.443095253477263 0loss/SRqqt2/detection_loss‚3.397613416699802 /loss/FBt3/detection_loss‚0.9842223143854807 0loss/SRqqt1/assignment_loss‚1.55988388727693 0loss/FBt4/assignment_loss‚0.2835947130732448 1loss/SRqqt4/detection_loss‚3.1374321149455175 /loss/FBt4/detection_loss‚1.5174433114762242 1loss/SRqqt3/detection_loss‚1.0163924062031915 .loss/FBt2/detection_loss‚2.365852190689607 _step‚221»²Lî]Ð× 0loss/FBt2/assignment_loss‚0.3388580597126065 /loss/FRt1/detection_loss‚2.6428627127950843 /loss/FBt1/detection_loss‚2.8984143882989883 _runtime‚3375.238352116 epoch‚2 % _timestamp‚1.7885323910849938e+09 0loss/FRt1/assignment_loss‚3.3957373824986545 /loss/FRt4/assignment_loss‚3.683554995059967 1loss/SRqqt3/assignment_loss‚1.065829457334158 .loss/FRt3/detection_loss‚1.443095253477263 0loss/SRqqt2/detection_loss‚3.397613416699802 /loss/FBt3/detection_loss‚0.9842223143854807 0loss/SRqqt1/assignment_loss‚1.55988388727693 0loss/FBt4/assignment_loss‚0.2835947130732448 1loss/SRqqt4/detection_loss‚3.1374321149455175 /loss/FBt4/detection_loss‚1.5174433114762242 1loss/SRqqt3/detection_loss‚1.0163924062031915 .loss/FBt2/detection_loss‚2.365852190689607 _step‚221 trainer/global_step‚2699 2loss/SRqqt4/assignment_loss‚1.7448205169704225 /loss/FRt2/detection_loss‚3.0564541128965526 /loss/FRt2/assignment_loss‚3.664677280646104 0loss/FBt1/assignment_loss‚0.2253778815660515 2loss/FBt3/assignment_loss‚0.010990529208562805 %loss/total_loss‚46.90986685889342 /loss/FRt3/assignment_loss‚1.743161570887233 /loss/FRt4/detection_loss‚2.9508929657936096 1loss/SRqqt1/detection_loss‚1.8986784979408862 2loss/SRqqt2/assignment_loss‚1.8840268990572762Ý‘(*‘Ñ‚(:† µ­ëÔÝᤚ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eý÷éqÒ‚(:æ µ­ëÔ¹£é disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.46484375 disk.sda.in‚0 disk.sda.out‚ 108.71875 disk.sda2.in‚0) disk./.usagePercent‚79.01757436424933' disk.nvme0c0n1.out‚66452.2392578125 disk.dm-0.in‚ 7877.328125" disk.dm-0.out‚66452.2392578125 disk.dm-1.out‚ 108.71875 disk.nvme0n1.in‚ 7877.328125 disk.sda3.in‚0) proc.memory.availableMB‚ 1214.15234375! disk.nvme0c0n1.in‚ 7877.328125% disk.nvme0n1.out‚66452.2392578125# memory_percent‚92.5894021987915# proc.memory.rssMB‚ 3043.94140625 cpu‚25.330421463277805 network.sent‚ 6.024968e+06 network.recv‚ 1.115173e+06 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 108.71875* proc.memory.percent‚18.578743934631348 proc.cpu.threads‚17$ disk./.usageGB‚1177.418960571289 disk.dm-1.in‚0—Ã~œÓ‚(:‘ Ä­ëÔ¸™Ê” gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚124.912' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.982400000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÐßòtÔ‚(:é Ä­ëÔÇëÏ— network.sent‚ 6.123837e+06 network.recv‚ 1.120324e+06' disk.nvme0c0n1.out‚66717.7978515625" disk.dm-0.out‚66717.7978515625 disk.sda2.in‚0 disk.sda3.out‚ 109.1015625# proc.memory.rssMB‚ 3043.94140625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1260.out‚ 0.46484375 disk.sda.out‚ 109.1015625 cpu‚25.329438353595) disk./.usagePercent‚78.98157023265637 disk.dm-0.in‚ 7877.328125 disk.dm-1.in‚0 disk.dm-1.out‚ 109.1015625 disk.drbd1260.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚66717.7978515625 disk.sda3.in‚0% disk./.usageGB‚1176.8824729919434! disk.nvme0c0n1.in‚ 7877.328125 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 7877.328125$ memory_percent‚92.58947372436523' proc.memory.availableMB‚ 1214.140625* proc.memory.percent‚18.578743934631348e—gz‘Õ‚(:† Ó­ëÔʹʕ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü {Ö‚(:ð Ó­ëÔúâ— disk.drbd1260.in‚0 disk.nvme0n1.in‚ 7877.328125$ memory_percent‚92.59264469146729 proc.cpu.threads‚17 network.sent‚ 6.128044e+06! disk.nvme0c0n1.in‚ 7877.328125 disk.dm-1.in‚0 disk.sda.out‚ 109.56640625 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1213.62109375% disk./.usageGB‚1175.9059562683105 disk.dm-0.in‚ 7877.328125# disk.dm-0.out‚67244.70947265625 disk.sda1.out‚0 disk.drbd1260.out‚0.46875 disk.sda.in‚0& disk.nvme0n1.out‚67244.70947265625 disk.sda3.in‚0 network.recv‚ 1.124781e+06) disk./.usagePercent‚78.91603537598127( disk.nvme0c0n1.out‚67244.70947265625 disk.dm-1.out‚ 109.56640625 disk.sda3.out‚ 109.56640625# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.32822338982134 disk.sda1.in‚0†­±2×- _wandbruntime‚3429  _runtime‚3429„Ó¦ÅØ¿ % _timestamp‚1.7885324454184625e+09 _runtime‚3429.572022944 _step‚222 % lr-AdamW/pg1‚0.000776911159613971 % lr-AdamW/pg2‚0.000776911159613971 trainer/global_step‚2749V ÊÙÄ % lr-AdamW/pg1‚0.000776911159613971 % lr-AdamW/pg2‚0.000776911159613971 trainer/global_step‚2749 % _timestamp‚1.7885324454184625e+09 _runtime‚3429.572022944 _step‚222ÞýráÏ2Ú- _wandbruntime‚3429  _runtime‚3429ت8‰+ÛÖO GADeviceStatsMonitor.on_train_batch¼ÆAã±'_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52206871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117303142336000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117303142336000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31186150 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31186152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5445054040899 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52206871 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124028832327168 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111842628491264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117303394007040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19825261 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124028519864320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7885324454194305e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5460765515776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111842610488320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111842628491264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51010726 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111887641752371 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106442604750832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52206873 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118473621986816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111887887772811 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111842610488320 _runtime‚3429.573287298 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2—Kb á'ÜÛO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111842628491264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117303394007040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19825261 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124028519864320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7885324454194305e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5460765515776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111842610488320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111842628491264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51010726 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111887641752371 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106442604750832 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52206873 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118473621986816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111887887772811 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111842610488320 _runtime‚3429.573287298 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52206871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117303142336000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117303142336000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31186150 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31186152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5445054040899 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52206871 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124028832327168 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712߉T*Ò2Ý- _wandbruntime‚3430  _runtime‚3430Ì,'æ&ÞàM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106480135108256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111881914528256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885324467328076e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126953298 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111927376619429 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111881896525312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 _runtime‚3430.887169578 _step‚224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126953298 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2749 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5462495873536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5462729554944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111927130598781 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74754521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5462729554944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74754521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52225146 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5556933499392 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117344644083200 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52225146 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111881896525312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19832513 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106480118068896 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52225148 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126979669 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117344392398848 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117344644083200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31196954 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74728152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111881914528256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124072530062848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126979669 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124072215516160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51029467 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51028783 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5447241511173 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118515305956864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118515282016768 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19831831 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5447012529885 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74754521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5462729554944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74754521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52225146 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5556933499392 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117344644083200 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52225146 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on)á¹Ó¸_train_batch_end/allocated_bytes.large_pool.freed‚111881896525312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19832513 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106480118068896 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52225148 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126979669 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117344392398848 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117344644083200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31196954 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74728152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111881914528256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124072530062848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126979669 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124072215516160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51029467 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51028783 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5447241511173 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118515305956864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118515282016768 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19831831 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5447012529885 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106480135108256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111881914528256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885324467328076e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126953298 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111927376619429 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111881896525312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 _runtime‚3430.887169578 _step‚224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126953298 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2749 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5462495873536àÍÂ×Þ2à- _wandbruntime‚3430  _runtime‚3430\(#‡Iáà 0loss/FRt4/assignment_loss‚3.6114470145919104 1loss/FBt4/assignment_loss‚0.18191048093147752 .loss/FRt1/detection_loss‚2.315519635493939 1loss/FBt2/assignment_loss‚0.22085719945825985 .loss/FRt3/detection_loss‚1.546767478529364 /loss/FBt2/detection_loss‚2.2706166982650755 /loss/FBt3/detection_loss‚0.9375726357684471 &loss/total_loss‚45.593771361807526 .loss/FRt2/detection_loss‚3.374695019721985 0loss/SRqqt4/detection_loss‚2.578280460089445 /loss/FRt1/assignment_loss‚4.327720321141756 /loss/FRt4/detection_loss‚2.8032474409450185 0loss/SRqqt2/detection_loss‚2.967596781881232 1loss/SRqqt3/detection_loss‚0.9389215546115389 % _timestamp‚1.7885324467354736e+09 _runtime‚3430.888865505 0loss/FRt2/assignment_loss‚3.5759032201766967 2loss/SRqqt3/assignment_loss‚1.0282410913902633 2loss/SRqqt4/assignment_loss‚1.9351080022752285 /loss/FBt1/detection_loss‚2.2587874811142683 /loss/FBt4/detection_loss‚1.5045410388059193 loss/FBt3/assignment_loss‚0 2loss/SRqqt1/assignment_loss‚1.5725676577538252 1loss/SRqqt2/assignment_loss‚1.823914683963123 0loss/FBt1/assignment_loss‚0.3060492341868155 epoch‚2 trainer/global_step‚2749 _step‚225 0loss/FRt3/assignment_loss‚1.5622236542403698 0loss/SRqqt1/detection_loss‚1.9512825764715671 NâÈ 2loss/SRqqt1/assignment_loss‚1.5725676577538252 1loss/SRqqt2/assignment_loss‚1.823914683963123 0loss/FBt1/assignment_loss‚0.3060492341868155 epoch‚2 trainer/global_step‚2749 _step‚225 0loss/FRt3/assignment_loss‚1.5622236542403698 0loss/SRqqt1/detection_loss‚1.951282576471567 0loss/FRt4/assignment_loss‚3.6114470145919104 1loss/FBt4/assignment_loss‚0.18191048093147752 .loss/FRt1/detection_loss‚2.315519635493939 1loss/FBt2/assignment_loss‚0.22085719945825985 .loss/FRt3/detection_loss‚1.546767478529364 /loss/FBt2/detection_loss‚2.2706166982650755 /loss/FBt3/detection_loss‚0.9375726357684471 &loss/total_loss‚45.593771361807526 .loss/FRt2/detection_loss‚3.374695019721985 0loss/SRqqt4/detection_loss‚2.578280460089445 /loss/FRt1/assignment_loss‚4.327720321141756 /loss/FRt4/detection_loss‚2.8032474409450185 0loss/SRqqt2/detection_loss‚2.967596781881232 1loss/SRqqt3/detection_loss‚0.9389215546115389 % _timestamp‚1.7885324467354736e+09 _runtime‚3430.888865505 0loss/FRt2/assignment_loss‚3.5759032201766967 2loss/SRqqt3/assignment_loss‚1.0282410913902633 2loss/SRqqt4/assignment_loss‚1.9351080022752285 /loss/FBt1/detection_loss‚2.2587874811142683 /loss/FBt4/detection_loss‚1.5045410388059193 loss/FBt3/assignment_loss‚0áÛŒ-@œã‚(:‘ â­ëÔ¬§ò• gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.488' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.897599999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…iò_ä‚(:Ô â­ëÔì´á– disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚8033.75 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚110.125 disk.sda2.out‚0 disk.nvme0n1.in‚8033.75& disk.nvme0n1.out‚67726.13916015625# disk.dm-0.out‚67726.13916015625 disk.dm-1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 1.128139e+06 disk.sda3.in‚0 disk.sda3.out‚110.125 disk.dm-0.in‚8033.75 disk.drbd1260.out‚0.46875# memory_percent‚92.5929307937622# proc.memory.rssMB‚ 3043.94140625 cpu‚25.327180160482367% disk./.usageGB‚1175.9053382873535( disk.nvme0c0n1.out‚67726.13916015625 disk.dm-1.out‚110.125) proc.memory.availableMB‚ 1213.57421875* proc.memory.percent‚18.578743934631348 network.sent‚ 6.13292e+06) disk./.usagePercent‚78.91599390276072Â@Í‘å‚(:† ñ­ëÔ”Û¶– gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚137.736' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.5472 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03O(_gæ‚(:Ü ñ­ëÔÍÆÉ— network.recv‚ 1.134716e+06) disk./.usagePercent‚78.91598801458744 disk.drbd1260.out‚ 0.47265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚8033.75$ memory_percent‚92.59002208709717 cpu‚25.325975423638766 network.sent‚ 6.241755e+06% disk./.usageGB‚1175.9052505493164 disk.dm-1.out‚110.6875 disk.drbd1260.in‚0 disk.sda.in‚0# disk.dm-0.out‚67930.89990234375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.out‚110.6875) proc.memory.availableMB‚ 1214.05078125# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 proc.cpu.threads‚17 disk.nvme0c0n1.in‚8033.75( disk.nvme0c0n1.out‚67930.89990234375 disk.dm-0.in‚8033.75 disk.sda1.out‚0 disk.sda.out‚110.6875& disk.nvme0n1.out‚67930.89990234375 disk.sda3.in‚0欃ç‚(:… €®ëÔÏÅò” gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.403' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8806 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã Üsè‚(:è €®ëÔï·Æ—* proc.memory.percent‚18.578743934631348 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 111.34765625 disk.sda2.in‚0 disk.sda3.out‚ 111.34765625$ memory_percent‚92.59307384490967# proc.memory.rssMB‚ 3043.94140625 cpu‚25.324935974111092) disk./.usagePercent‚78.91761955060315 disk.dm-0.in‚8033.75# disk.dm-0.out‚68080.32177734375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚68080.32177734375 network.recv‚ 1.138251e+06% disk./.usageGB‚1175.9295616149902 disk.nvme0c0n1.in‚8033.75( disk.nvme0c0n1.out‚68080.32177734375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.47265625 disk.nvme0n1.in‚8033.75 disk.sda3.in‚0 network.sent‚ 6.246671e+06 disk.dm-1.out‚ 111.34765625) proc.memory.availableMB‚ 1213.55078125k¿¯‘é‚(:† ®ëÔ¬ÿ«– gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚126.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.3276 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï’&–gê‚(:Ü ®ëÔÚÊ™( disk./.usagePercent‚78.9198002227798 disk.drbd1260.out‚ 0.4765625 disk.sda.out‚ 111.69140625 disk.nvme0n1.in‚8033.75$ disk.nvme0n1.out‚68218.376953125 disk.sda3.out‚ 111.69140625 disk.dm-0.in‚8033.75! disk.dm-0.out‚68218.376953125 network.recv‚ 1.142702e+06 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3043.94140625 network.sent‚ 6.25281e+06 disk.sda2.in‚0& disk.nvme0c0n1.out‚68218.376953125 disk.dm-1.out‚ 111.69140625 disk.sda1.in‚0' proc.memory.availableMB‚ 1214.046875* proc.memory.percent‚18.578743934631348 cpu‚25.32387543250028% disk./.usageGB‚1175.9620552062988 disk.nvme0c0n1.in‚8033.75$ memory_percent‚92.59004592895508 proc.cpu.threads‚17Yq¬2ë- _wandbruntime‚3482  _runtime‚3482F ì’ÇìÁ & lr-AdamW/pg1‚0.0007743375189043651 & lr-AdamW/pg2‚0.0007743375189043651 trainer/global_step‚2799 % _timestamp‚1.7885324985981455e+09 _runtime‚3482.751598002 _step‚226½Öp"ÌíÆ & lr-AdamW/pg1‚0.0007743375189043651 & lr-AdamW/pg2‚0.0007743375189043651 trainer/global_step‚2799 % _timestamp‚1.7885324985981455e+09 _runtime‚3482.751598002 _step‚226âÅÚs2î- _wandbruntime‚3482  _runtime‚3482;‚ÐÚ'ïÔO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3482.752661888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53120623 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51902242 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31726252 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5654776351232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108319122622032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108319105582672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20175291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129160033 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20175990 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5543221062487 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53120621 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76065755 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119365911563264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119365659892224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5558747554304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5558747554304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113806930340864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113862343684519 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129186378 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 # _timestamp‚1.78853249859906e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113806930340864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76039412 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126212572900352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5655066971136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113862343684519 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129186378 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 # _timestamp‚1.78853249859906e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool˜æ.allocated‚113806930340864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76039412 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126212572900352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5655066971136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119365659892224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5558747554304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5558747554304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136ãˆB2ñ- _wandbruntime‚3483  _runtime‚3483.s±æ&òàM _step‚228 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120599504459264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚2799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76066104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51919040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51919719 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5545179551681 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113846198374912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5560945261568 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53138896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126256269928448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108356635940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120599480519168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53138898 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76066104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5544950570393 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53138898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31737054 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113901832531137 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129231371 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5560711580160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53138896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119406909955072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113901586510489 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113846198374912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7885324995935836e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129205000 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76092473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5560711580160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5560945261568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20182665 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108356652979456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273d™{ë&óåM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129231371 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129205000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76092473 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5657080015872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113846216377856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5656789409280 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76066104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51919040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51919719 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5545179551681 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113846198374912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5560945261568 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53138896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126256269928448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108356635940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120599480519168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53138898 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76066104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5544950570393 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53138898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31737054 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113901832531137 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129231371 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5560711580160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53138896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119406909955072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113901586510489 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113846198374912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7885324995935836e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129205000 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76092473 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5560711580160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5560945261568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20182665 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108356652979456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚228 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120599504459264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚2799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E-rø‚(:ç ž®ëÔÇ‹ø¤ cpu‚25.322824196983397) disk./.usagePercent‚78.91925748680721 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4765625 disk.sda.in‚0$ disk.nvme0n1.out‚68515.060546875 disk.sda3.out‚ 112.34375 network.recv‚ 1.145838e+06 disk.dm-1.out‚ 112.34375 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 8165.4921875 disk.dm-0.in‚ 8165.4921875$ memory_percent‚92.59331226348877 proc.cpu.threads‚17 network.sent‚ 6.257632e+06 disk.sda1.in‚0 disk.nvme0n1.in‚ 8165.4921875 disk.sda3.in‚0& disk.nvme0c0n1.out‚68515.060546875! disk.dm-0.out‚68515.060546875) proc.memory.availableMB‚ 1213.51171875* proc.memory.percent‚18.578743934631348% disk./.usageGB‚1175.9539680480957 disk.sda.out‚ 112.34375 disk.sda2.in‚0# proc.memory.rssMB‚ 3043.94140625Öçš`‘ù‚(:† ­®ëԉݡ• gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚100.787' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.1574 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äýUGú‚(:ô ­®ëÔîç¹—% disk./.usageGB‚1175.9422492980957 disk.dm-1.out‚ 112.87109375 disk.sda1.in‚0& disk.nvme0n1.out‚68683.87255859375# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.32180102042646 proc.cpu.threads‚17 network.recv‚ 1.152869e+06 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 112.87109375" disk.nvme0c0n1.in‚ 8207.4921875( disk.nvme0c0n1.out‚68683.87255859375) disk./.usagePercent‚78.91847103166202 disk.dm-0.in‚ 8207.4921875# disk.dm-0.out‚68683.87255859375 disk.drbd1260.out‚ 0.48046875 disk.sda.in‚0 disk.sda.out‚ 112.87109375 disk.nvme0n1.in‚ 8207.4921875$ memory_percent‚92.59328842163086 network.sent‚ 6.368443e+06 disk.dm-1.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 1213.515625¼ëï±û‚(:… ¼®ëÔ…ź˜ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.933 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º%·ü‚(:ö ¼®ëÔôž ) disk./.usagePercent‚78.92687115087553 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 3043.94140625 proc.cpu.threads‚17 network.sent‚ 6.371573e+06 disk.sda3.out‚ 113.39453125" disk.nvme0c0n1.in‚ 8207.4921875 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚92.59188175201416) proc.memory.availableMB‚ 1213.74609375 cpu‚25.32078195219922 network.recv‚ 1.156297e+06% disk./.usageGB‚1176.0674171447754 disk.nvme0n1.in‚ 8207.4921875 disk.sda3.in‚0( disk.nvme0c0n1.out‚69213.82568359375 disk.dm-0.in‚ 8207.4921875# disk.dm-0.out‚69213.82568359375 disk.dm-1.out‚ 113.39453125& disk.nvme0n1.out‚69213.82568359375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.48046875 disk.sda.out‚ 113.39453125* proc.memory.percent‚18.578743934631348)Rʽœý‚(:‘ Ë®ëÔŠºîš gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.504' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.300800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ΰyþ‚(:î Ë®ëÔ¯¼—¢ disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd1260.out‚0.484375 disk.sda2.out‚0 network.recv‚ 1.161081e+06) disk./.usagePercent‚78.93681832361843% disk.nvme0n1.out‚69399.9970703125 disk.sda3.out‚ 113.8515625' disk.nvme0c0n1.out‚69399.9970703125 disk.dm-1.out‚ 113.8515625 disk.sda1.out‚0 disk.sda2.in‚0% disk./.usageGB‚1176.2156372070312 disk.sda.in‚0$ memory_percent‚92.59495735168457( proc.memory.availableMB‚ 1213.2421875 disk.nvme0n1.in‚ 8207.4921875" disk.nvme0c0n1.in‚ 8207.4921875 disk.drbd1260.in‚0 disk.sda.out‚ 113.8515625# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.320062752341617 proc.cpu.threads‚17 network.sent‚ 6.379543e+06 disk.sda3.in‚0 disk.dm-0.in‚ 8207.4921875" disk.dm-0.out‚69399.9970703125=¬©Í2ÿ- _wandbruntime‚3540  _runtime‚3540¾Ý/Æ€À trainer/global_step‚2849 $ _timestamp‚1.788532556149438e+09 _runtime‚3540.303100236 _step‚230 & lr-AdamW/pg1‚0.0007716875977033286 & lr-AdamW/pg2‚0.0007716875977033286H®`çËÅ trainer/global_step‚2849 $ _timestamp‚1.788532556149438e+09 _runtime‚3540.303100236 _step‚230 & lr-AdamW/pg1‚0.0007716875977033286 & lr-AdamW/pg2‚0.0007716875977033286æ’­‰ó2‚- _wandbruntime‚3540  _runtime‚3540Ü@úªƒÕO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131411735 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_srüZ1&tart/inactive_split.all.current‚704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110195640493232 $ _timestamp‚1.788532556150883e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54034371 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121428177448448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115771214187520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5657196929024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77403707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77377364 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121428429119488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121428177448448 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115771232190464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131438080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115771232190464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122641995051520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52788508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20521454 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77377364 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54034371 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5754883516416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5657196929024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5641159102995 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32266350`ñï¤à'„ÚO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20521454 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77377364 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54034371 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5754883516416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5657196929024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5641159102995 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32266350 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131411735 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110195640493232 $ _timestamp‚1.788532556150883e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54034371 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121428177448448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115771214187520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5657196929024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77403707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77377364 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121428429119488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121428177448448 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115771232190464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131438080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115771232190464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122641995051520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52788508ç¹gƒû2…- _wandbruntime‚3541  _runtime‚3541ùêæ&†àM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5659160968192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110233153811296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32277152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77404056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131483073 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128440584693248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5756584833536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5658927286784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52805162 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5643117592189 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115810518227456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131456702 _step‚232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131483073 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54052646 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77430425 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54052646 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77430425 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121469427511296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52805871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32277154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122683679021568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54052648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115810500224512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115810500224512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115876288442845 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚3541.408603007 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20528010I…R´ ‡åM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121469679195648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885325572537978e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77404056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131456702 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121469679195648 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5658927286784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5642888610901 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54052648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115810500224512 ZFDeviceStatsMonitor.on_train_batch_end/1ÿCE7allocated_bytes.large_pool.freed‚115810500224512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115876288442845 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚3541.408603007 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20528010 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5659160968192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110233153811296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32277152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77404056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131483073 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128440584693248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5756584833536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5658927286784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52805162 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5643117592189 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115810518227456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131456702 _step‚232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131483073 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54052646 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77430425 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54052646 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77430425 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121469427511296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52805871 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32277154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122683679021568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53674668 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53673971 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 trainer/global_step‚2899 _runtime‚3596.709817455 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5755412635648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54948123 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124726193553920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117811009487495 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123490695004672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5854508611072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5755178967552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78715316 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚235 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78741659 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78741659 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5755178967552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32806450 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112072141325072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117811255507935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5739097143503 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117735534040064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32806452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5854795036672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117735534040064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20868216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20867521 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123490695004672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54948121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112072158364432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130581012530688 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133663437 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133689782 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54948121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124726217494016fÿE€^–ÛO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53674668 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53673971 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 trainer/global_step‚2899 _runtime‚3596.709817455 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5755412635648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54948123 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124726193553920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117811009487495 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123490695004672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5854508611072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5755178967552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78715316 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚235 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78741659 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78741659 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5755178967552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32806450 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112072141325072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117811255507935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5739097143503 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117735534040064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32806452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5854795036672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117735534040064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20868216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20867521 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123490695004672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54948121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112072158364432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130581012530688 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133663437 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133689782 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54948121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124726217494016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78742008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54966398 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5757376674816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133734775 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5757142993408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123532196751872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32817254 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54966396 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117774802074112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53691728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32817252 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124767901464064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5856545540608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78768377 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚236 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117850498333905 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117774802074112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133734775 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124767877523968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78768377 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20875178 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117850744354553 trainer/global_step‚2899 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117774820077056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944pRóë&™åM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78742008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54966398 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5757376674816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133734775 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5757142993408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123532196751872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32817254 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54966396 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117774802074112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53691728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32817252 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124767901464064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5856545540608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78768377 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚236 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117850498333905 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117774802074112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133734775 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124767877523968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78768377 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20875178 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117850744354553 trainer/global_step‚2899 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117774820077056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123531945067520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384ìßߎ¨2š- _wandbruntime‚3597  _runtime‚3597øñÁV›Ð _runtime‚3597.688099841 1loss/SRqqt2/assignment_loss‚2.026843730774191 0loss/FBt1/assignment_loss‚0.2851378233127114 .loss/FRt2/detection_loss‚2.904901360020493 /loss/FRt4/detection_loss‚3.0446670093844013 -loss/FBt4/detection_loss‚1.09374838079824 0loss/FRt3/assignment_loss‚1.4021415948867797 /loss/FRt4/assignment_loss‚3.363353233183584 1loss/FBt4/assignment_loss‚0.29381601087877346 1loss/SRqqt4/detection_loss‚2.7342288891474404 /loss/FBt2/detection_loss‚3.3805350065231323 -loss/FBt3/detection_loss‚1.32520542293787 1loss/SRqqt4/assignment_loss‚1.611321133162294 1loss/FBt2/assignment_loss‚0.12063405778371816 .loss/FRt1/detection_loss‚2.243365521493711 /loss/FBt1/detection_loss‚1.9083660620230216 _step‚237 %loss/total_loss‚45.16334089077169 epoch‚2 .loss/FRt2/assignment_loss‚4.17319076350241 2loss/SRqqt1/assignment_loss‚1.8147372392316659 2loss/SRqqt3/assignment_loss‚1.0203322581787648 /loss/FRt3/detection_loss‚1.4225668612453672 1loss/SRqqt1/detection_loss‚2.0409031634529433 1loss/SRqqt3/detection_loss‚0.7591050395216697 /loss/FRt1/assignment_loss‚3.574607516589918 0loss/SRqqt2/detection_loss‚2.600152046592147 trainer/global_step‚2899 % _timestamp‚1.7885326135347388e+09 2loss/FBt3/assignment_loss‚0.019480766146443784˜â…>[œÕ /loss/FRt3/detection_loss‚1.4225668612453672 1loss/SRqqt1/detection_loss‚2.0409031634529433 1loss/SRqqt3/detection_loss‚0.7591050395216697 /loss/FRt1/assignment_loss‚3.574607516589918 0loss/SRqqt2/detection_loss‚2.600152046592147 trainer/global_step‚2899 % _timestamp‚1.7885326135347388e+09 2loss/FBt3/assignment_loss‚0.019480766146443784 _runtime‚3597.688099841 1loss/SRqqt2/assignment_loss‚2.026843730774191 0loss/FBt1/assignment_loss‚0.2851378233127114 .loss/FRt2/detection_loss‚2.904901360020493 /loss/FRt4/detection_loss‚3.0446670093844013 -loss/FBt4/detection_loss‚1.09374838079824 0loss/FRt3/assignment_loss‚1.4021415948867797 /loss/FRt4/assignment_loss‚3.363353233183584 1loss/FBt4/assignment_loss‚0.29381601087877346 1loss/SRqqt4/detection_loss‚2.7342288891474404 /loss/FBt2/detection_loss‚3.3805350065231323 -loss/FBt3/detection_loss‚1.32520542293787 1loss/SRqqt4/assignment_loss‚1.611321133162294 1loss/FBt2/assignment_loss‚0.12063405778371816 .loss/FRt1/detection_loss‚2.243365521493711 /loss/FBt1/detection_loss‚1.9083660620230216 _step‚237 %loss/total_loss‚45.16334089077169 epoch‚2 .loss/FRt2/assignment_loss‚4.17319076350241 2loss/SRqqt1/assignment_loss‚1.8147372392316659 2loss/SRqqt3/assignment_loss‚1.0203322581787648íkZlEœ‚(:‘ ‡¯ëÔËäó” gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚137.398' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.479599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×? vž‚(:ë ‡¯ëÔ滞— disk.dm-1.out‚ 115.44921875 disk.drbd1260.in‚0 disk.sda.out‚ 115.44921875# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.316382849752443 proc.cpu.threads‚17% disk./.usageGB‚1176.3092727661133 disk.sda3.in‚0 disk.sda3.out‚ 115.44921875$ disk.nvme0c0n1.out‚ 70546.5390625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚92.59543418884277 network.recv‚ 1.18558e+06) disk./.usagePercent‚78.94310228455389# disk.nvme0c0n1.in‚ 8323.42578125 disk.dm-0.in‚ 8323.42578125 disk.dm-0.out‚ 70546.5390625 disk.sda1.in‚0 disk.drbd1260.out‚ 0.4921875 disk.sda2.in‚0 network.sent‚ 6.505332e+06 disk.sda1.out‚0! disk.nvme0n1.in‚ 8323.42578125" disk.nvme0n1.out‚ 70546.5390625( proc.memory.availableMB‚ 1213.1640625(Ù•œŸ‚(:‘ –¯ëÔ…ëÖ• gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚133.301' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.660199999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VýàØx ‚(:í –¯ëÔÂÅâ˜% disk./.usageGB‚1176.3093223571777 disk.drbd1260.out‚ 0.4921875 disk.sda.in‚0 disk.sda3.out‚ 115.890625 disk.drbd1260.in‚0 network.sent‚ 6.613121e+06 disk.dm-0.in‚ 8323.42578125# disk.dm-0.out‚70828.88525390625 disk.dm-1.out‚ 115.890625 disk.sda.out‚ 115.890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 8323.42578125 disk.sda3.in‚0$ proc.memory.availableMB‚1213.625# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348) disk./.usagePercent‚78.94310561265183# disk.nvme0c0n1.in‚ 8323.42578125( disk.nvme0c0n1.out‚70828.88525390625 disk.sda1.in‚0& disk.nvme0n1.out‚70828.88525390625$ memory_percent‚92.59262084960938 cpu‚25.31526659012944 proc.cpu.threads‚17 network.recv‚ 1.190762e+06›ïd‘¡‚(:† ¥¯ëÔܲù– gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚139.569' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9138 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00»ªa~¢‚(:ó ¥¯ëÔÉ—ýš) disk./.usagePercent‚78.94310561265183& disk.nvme0n1.out‚71022.92138671875 disk.sda3.in‚0 disk.sda3.out‚ 116.265625( disk.nvme0c0n1.out‚71022.92138671875 disk.dm-0.in‚ 8439.58984375# disk.dm-0.out‚71022.92138671875 disk.sda.in‚0 network.sent‚ 6.620278e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 116.265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚92.59254932403564% disk./.usageGB‚1176.3093223571777# disk.nvme0c0n1.in‚ 8439.58984375 disk.drbd1260.out‚ 0.49609375) proc.memory.availableMB‚ 1213.63671875# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.31435456132519 proc.cpu.threads‚17 network.recv‚ 1.196918e+06! disk.nvme0n1.in‚ 8439.58984375 disk.dm-1.in‚0 disk.sda.out‚ 116.265625tøT‘£‚(:† ´¯ëÔ·äà” gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7546 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È´žã„¤‚(:ù ´¯ëÔóÍí— network.recv‚ 1.206894e+06 disk.dm-0.in‚ 8439.58984375! disk.nvme0n1.in‚ 8439.58984375( disk.nvme0c0n1.out‚71172.93310546875) proc.memory.availableMB‚ 1213.39453125 cpu‚25.31332152996465 proc.cpu.threads‚17) disk./.usagePercent‚78.94154396669298 disk.drbd1260.in‚0 disk.sda.out‚ 116.69921875 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚92.59402751922607* proc.memory.percent‚18.578743934631348 disk.dm-1.in‚0 disk.dm-1.out‚ 116.69921875 disk.sda.in‚0& disk.nvme0n1.out‚71172.93310546875 disk.sda3.out‚ 116.69921875# proc.memory.rssMB‚ 3043.94140625 network.sent‚ 6.628448e+06% disk./.usageGB‚1176.2860527038574# disk.dm-0.out‚71172.93310546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.49609375 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 8439.58984375›”³¹2¥- _wandbruntime‚3649  _runtime‚3649DLòCǦÁ & lr-AdamW/pg1‚0.0007661611385670445 & lr-AdamW/pg2‚0.0007661611385670445 trainer/global_step‚2949 % _timestamp‚1.7885326651755977e+09 _runtime‚3649.329069751 _step‚238dos¿Ì§Æ % _timestamp‚1.7885326651755977e+09 _runtime‚3649.329069751 _step‚238 & lr-AdamW/pg1‚0.0007661611385670445 & lr-AdamW/pg2‚0.0007661611385670445 trainer/global_step‚2949î`ÿÇ2¨- _wandbruntime‚3649  _runtime‚3649‘4AÊ©ÖO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132765169698816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80079611 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54561845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21214627 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125553212560896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119699817886720 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80053268 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125553212560896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_XG¦˜ bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125553464231936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚666 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853628342272 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33346552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126810415996416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885326651765885e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55861873 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5853394674176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33346550 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5954469373952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚2949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125553464231936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80053268 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113948659196272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119699817886720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21215293 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132764861430272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135941484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33346552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126810415996416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885326651765885e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55861873 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5853394674176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33346550 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5954469373952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚2949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125553464231936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80053268 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113948659196272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119699817886720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21215293 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132764861430272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135941484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132765169698816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80079611 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54561845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21214627 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125553212560896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119699817886720 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80053268 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125553212560896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125553464231936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚666 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853628342272 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135986477 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55880148 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80106329 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135960106 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54578917 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55880148 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55880146 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132808541869568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5956748061184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125594462623744 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80079960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125594714308096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119824954245613 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55880146 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33357354 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125594714308096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5838993673205 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119825200266261 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885326662080297e+09 _step‚240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135960106 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33357352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5956465843200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80079960 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2949 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119739121926656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21221565 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582ÖH{@Ò­åM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5855592381440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119739121926656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113986206593056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119739103923712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125594462623744 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80079960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125594714308096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119824954245613 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55880146 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33357354 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125594714308096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5838993673205 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119825200266261 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885326662080297e+09 _step‚240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135960106 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33357352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5956465843200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_poIæÙWol.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80079960 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2949 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119739121926656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21221565 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5855358700032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _runtime‚3650.362234798 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135986477 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55880148 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80106329 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135960106 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54578917 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55880148 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55880146 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132808541869568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5956748061184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934973224519 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127615730117120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121664137739264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55446486 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5934744243439 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56775621 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127615730117120 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138193186 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128894590558720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115825177067472 trainer/global_step‚2999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21560538 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33886650 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134948990576128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 _runtime‚3700.961290902 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚115825194106832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5951610380800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127615981788160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81391220 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 # _timestamp‚1.78853271680755e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56775621 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138166841 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134949298844672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56775623 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55447190 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81391220 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5951844048896 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5951610380800 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121664119736320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128894614498816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21559836 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127615981788160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5951610380800 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121664119736320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128894614498816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21559836 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127615981788160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934973224519 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127615730117120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121664137739264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55446486 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5934744243439 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56775621 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127615730117120 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138193186 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128894590558720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115825177067472 trainer/global_step‚2999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21560538 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33886650 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134948990576128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 _runtime‚3700.961290902 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚115825194106832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5951610380800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127615981788160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81391220 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 # _timestamp‚1.78853271680755e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56775621 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_baÞåÀAtch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138166841 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134949298844672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309ó@¨4E2½- _wandbruntime‚3702  _runtime‚3702Š…ä&¾ÞM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81417912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56793896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21566517 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128936298468864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115862724464256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6056683611648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138211808 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127657231864320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121799410157321 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5936702732425 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56793898 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121703405773312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5936931713713 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138238179 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81417912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121703423776256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81444281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33897452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 327129600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33897454 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5953808088064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121799656177969 _runtime‚ 3702.00775699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127657231864320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚244 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5953574406656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134992654950912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6056380422144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55463969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121703423776256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138238179 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21567196 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56793896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121703405773312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81444281 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128936274528768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56793896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121703405773312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81444281 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128936274528768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6056683611648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138211808 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127657231864320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121799410157321 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5936702732425 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56793898 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121703405773312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5936931713713 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138238179 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81417912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121703423776256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81444281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33897452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 327129600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33897454 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5953808088064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121799656177969 _runtime‚ 3702.00775699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127657231864320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚244 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5953574406656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134992654950912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6056380422144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55463969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121703423776256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138238179 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21567196 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309ô3E¥F2À- _wandbruntime‚3702  _runtime‚3702Ìà{ŽWÁÑ /loss/FBt4/assignment_loss‚0.248364879197174 1loss/SRqqt4/detection_loss‚2.4236280161600847 1loss/SRqqt3/detection_loss‚0.8943241126214464 .loss/FBt1/detection_loss‚2.916983480254809 epoch‚2 1loss/FBt1/assignment_loss‚0.34414923798647123 .loss/FRt3/detection_loss‚1.595931562913082 /loss/FBt2/detection_loss‚2.0117754131555556 0loss/SRqqt2/assignment_loss‚2.43341861558812 0loss/SRqqt4/assignment_loss‚1.98868486772363 3loss/FBt3/assignment_loss‚4.248570200173306e-05 .loss/FRt1/detection_loss‚2.395473856065008 /loss/FRt4/detection_loss‚3.1992286145687103 %loss/total_loss‚47.75217961927957 2loss/SRqqt1/assignment_loss‚2.1228713269640758 1loss/FBt2/assignment_loss‚0.38681650334510775 trainer/global_step‚2999 % _timestamp‚1.7885327178560276e+09 0loss/FRt3/assignment_loss‚1.5846472615148963 0loss/SRqqt1/detection_loss‚2.149960184969553 .loss/FBt4/detection_loss‚1.113325873259888 _step‚245 /loss/FRt1/assignment_loss‚4.370931560794513 0loss/FRt4/assignment_loss‚3.6112003127733865 .loss/FRt2/detection_loss‚2.988871437807878 _runtime‚3702.009703568 /loss/FRt2/assignment_loss‚4.059055772920449 2loss/SRqqt3/assignment_loss‚1.0183170747632782 1loss/SRqqt2/detection_loss‚2.5437890461512973 /loss/FBt3/detection_loss‚1.35038812207915878"1Œ\ÂÖ 0loss/SRqqt2/assignment_loss‚2.43341861558812 0loss/SRqqt4/assignment_loss‚1.98868486772363 3loss/FBt3/assignment_loss‚4.248570200173306e-05 .loss/FRt1/detection_loss‚2.395473856065008 /loss/FRt4/detection_loss‚3.1992286145687103 %loss/total_loss‚47.75217961927957 2loss/SRqqt1/assignment_loss‚2.1228713269640758 1loss/FBt2/assignment_loss‚0.38681650334510775 trainer/global_step‚2999 % _timestamp‚1.7885327178560276e+09 0loss/FRt3/assignment_loss‚1.5846472615148963 0loss/SRqqt1/detection_loss‚2.149960184969553 .loss/FBt4/detection_loss‚1.113325873259888 _step‚245 /loss/FRt1/assignment_loss‚4.370931560794513 0loss/FRt4/assignment_loss‚3.6112003127733865 .loss/FRt2/detection_loss‚2.988871437807878 _runtime‚3702.009703568 /loss/FRt2/assignment_loss‚4.059055772920449 2loss/SRqqt3/assignment_loss‚1.0183170747632782 1loss/SRqqt2/detection_loss‚2.5437890461512973 /loss/FBt3/detection_loss‚1.3503881220791587 /loss/FBt4/assignment_loss‚0.248364879197174 1loss/SRqqt4/detection_loss‚2.4236280161600847 1loss/SRqqt3/detection_loss‚0.8943241126214464 .loss/FBt1/detection_loss‚2.916983480254809 epoch‚2 1loss/FBt1/assignment_loss‚0.34414923798647123 .loss/FRt3/detection_loss‚1.595931562913082 /loss/FBt2/detection_loss‚2.0117754131555556õ¨ß‘Â(:† ð¯ëÔ—¨Ð” gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.129' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'`D}Ä‚(:ò ð¯ëÔÊŠ½— disk.dm-1.out‚ 118.3046875 disk.drbd1260.out‚ 0.50390625 disk.sda.in‚0 disk.sda2.out‚0 network.recv‚ 1.245508e+06 disk.sda3.in‚0 disk.sda3.out‚ 118.3046875 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 8602.6015625& disk.nvme0n1.out‚72589.79931640625) proc.memory.availableMB‚ 1212.58984375$ disk./.usageGB‚1176.309555053711 disk.sda1.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.30930422412613) disk./.usagePercent‚78.94312122911143" disk.nvme0c0n1.in‚ 8602.6015625( disk.nvme0c0n1.out‚72589.79931640625 disk.sda.out‚ 118.3046875$ memory_percent‚92.59893894195557 proc.cpu.threads‚17 network.sent‚ 6.764286e+06 disk.dm-0.in‚ 8602.6015625# disk.dm-0.out‚72589.79931640625 disk.dm-1.in‚0Nò=‘Å‚(:† ÿ¯ëÔ¡üü• gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óž#qÆ‚(:æ ÿ¯ëÔ­¥°˜% disk./.usageGB‚1175.7491455078125 disk.sda2.in‚0% disk.nvme0n1.out‚72813.8447265625" disk.nvme0c0n1.in‚ 8602.6015625 disk.dm-0.in‚ 8602.6015625 disk.sda1.out‚0$ memory_percent‚92.60518550872803) disk./.usagePercent‚78.90551167427125 disk.sda.in‚0 disk.sda.out‚118.8125 disk.sda3.out‚118.8125" disk.dm-0.out‚72813.8447265625 disk.drbd1260.out‚ 0.5078125# proc.memory.rssMB‚ 3043.94140625 proc.cpu.threads‚17 network.sent‚ 6.882909e+06 disk.sda3.in‚0' disk.nvme0c0n1.out‚72813.8447265625 disk.dm-1.in‚0) proc.memory.availableMB‚ 1211.56640625 disk.sda2.out‚0 disk.nvme0n1.in‚ 8602.6015625 disk.dm-1.out‚118.8125 disk.sda1.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚18.578743934631348 cpu‚25.30838891026983 network.recv‚ 1.271723e+06ór{œÇ‚(:‘ Ž°ëÔ¯¥Ë• gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.791000000000004 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘)ô‚xÈ‚(:í Ž°ëÔòÈØ—$ memory_percent‚92.59634017944336' proc.memory.availableMB‚ 1213.015625# proc.memory.rssMB‚ 3043.94140625 proc.cpu.threads‚17 network.recv‚ 1.275298e+06( disk./.usagePercent‚78.9201486490339% disk./.usageGB‚1175.9672470092773 disk.sda2.in‚0% disk.nvme0n1.out‚73009.8447265625" disk.nvme0c0n1.in‚ 8602.6015625 disk.dm-0.in‚ 8602.6015625 disk.sda2.out‚0 disk.nvme0n1.in‚ 8602.6015625' disk.nvme0c0n1.out‚73009.8447265625 disk.sda1.out‚0 disk.drbd1260.out‚ 0.5078125 disk.sda.out‚ 119.1640625 cpu‚25.307437159963673 disk.dm-1.in‚0 disk.dm-1.out‚ 119.1640625 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0* proc.memory.percent‚18.578743934631348 network.sent‚ 6.887789e+06 disk.sda3.in‚0 disk.sda3.out‚ 119.1640625" disk.dm-0.out‚73009.8447265625rÿq8‘É‚(:† °ëÔÐÎò– gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f§ìÍ‚Ê‚(:÷ °ëÔ³±Ù› disk.sda3.in‚0 disk.sda3.out‚ 119.69140625# disk.nvme0c0n1.in‚ 8738.93359375 disk.dm-1.in‚0 disk.dm-1.out‚ 119.69140625 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 8738.93359375% disk.nvme0n1.out‚73522.6162109375' disk.nvme0c0n1.out‚73522.6162109375 disk.dm-0.in‚ 8738.93359375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.51171875 network.sent‚ 6.897647e+06 network.recv‚ 1.286624e+06% disk./.usageGB‚1175.9901657104492" disk.dm-0.out‚73522.6162109375) proc.memory.availableMB‚ 1212.98046875) disk./.usagePercent‚78.92168674229963 disk.sda2.out‚0$ memory_percent‚92.59655475616455# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.306626258152335 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 119.69140625¹ê|)2Ë- _wandbruntime‚3760  _runtime‚3760…¥GÅÌ¿ % _timestamp‚1.7885327759473827e+09 _runtime‚3760.100947879 _step‚246 % lr-AdamW/pg1‚0.000760336388788308 % lr-AdamW/pg2‚0.000760336388788308 trainer/global_step‚3049|ùuÊÍÄ % _timestamp‚1.7885327759473827e+09 _runtime‚3760.100947879 _step‚246 % lr-AdamW/pg1‚0.000760336388788308 % lr-AdamW/pg2‚0.000760336388788308 trainer/global_step‚3049ö%s×ö2Î- _wandbruntime‚3760  _runtime‚3760Vô_%ÏÖO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6050059755520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82755515 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6032911265027 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57689373 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21907577 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130978813001216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117701694938672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57689371 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137133159751168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34426752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129678247673344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3049 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 % _timestamp‚1.7885327759485197e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129678499344384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130978789061120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129678247673344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123628439588864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6049826087424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140444888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123628439588864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117701711978032÷ûá'ÐÛO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82755515 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57689371 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57689373 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123628421585920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56333635 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117701711978032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6050059755520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82755515 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6032911265027 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57689373 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21907577 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130978813001216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117701694938672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57689371 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137133159751168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34426752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129678247673344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3049 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 % _timestamp‚1.7885327759485197e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129678499344384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130978789061120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129678247673344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123628439588864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6049826087424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140444888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123628439588864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57707648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123667725625856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56351496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6156375304704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129719749420544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34437552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131020496971264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117739225296096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140489881 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3761.189910036 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6051790113280 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82782233 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140489881 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140463510 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129719749420544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21914609 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 $ _timestamp‚1.788532777035438e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123774112089677 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123667707622912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123667707622912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34437554 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21913944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123773866069029 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129719497736192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6052023794688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚3049 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57707648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6156651231232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82755864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0s\‚.ÓäM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123667707622912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123667707622912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34437554 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21913944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123773866069029 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129719497736192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6052023794688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚3049 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57707648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6156651231232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82755864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57707646 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82782233 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6052023794688 _step‚248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137177148202496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131020473031168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57707648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123667725625856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56351496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6156375304704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129719749420544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34437552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.currÕòmô¼ent‚665 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131020496971264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117739225296096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140489881 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3761.189910036 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6051790113280 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82782233 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140489881 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140463510 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129719749420544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21914609 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 $ _timestamp‚1.788532777035438e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123774112089677 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408øñ‘.2Ô- _wandbruntime‚3761  _runtime‚3761»O™eYÕÓ /loss/FRt3/detection_loss‚1.3954136008281766 1loss/SRqqt1/detection_loss‚1.9854332864284516 % _timestamp‚1.7885327770387433e+09 0loss/FRt3/assignment_loss‚1.4214408611073905 /loss/FBt2/detection_loss‚2.4321776470717262 /loss/FBt4/detection_loss‚1.1996294883164493 /loss/FRt1/assignment_loss‚4.809950606822968 2loss/SRqqt2/assignment_loss‚1.8562748270301983 0loss/SRqqt4/detection_loss‚2.969901767373085 %loss/total_loss‚45.22396721881279 epoch‚2 1loss/FBt3/assignment_loss‚0.02757631320520503 1loss/FBt4/assignment_loss‚0.24495806341399823 .loss/FBt1/detection_loss‚2.145949512720108 /loss/FBt3/detection_loss‚0.9610073503695036 _runtime‚3761.192181044 2loss/SRqqt1/assignment_loss‚1.5900925731286406 2loss/SRqqt4/assignment_loss‚1.9627522885799409 0loss/FBt2/assignment_loss‚0.3065152351011741 1loss/SRqqt2/detection_loss‚2.6653938910056807 trainer/global_step‚3049 /loss/FRt2/assignment_loss‚3.847057254083695 _step‚249 0loss/FRt4/assignment_loss‚3.1928676495680937 2loss/SRqqt3/assignment_loss‚0.9920226504643247 .loss/FRt4/detection_loss‚2.766333725001361 0loss/SRqqt3/detection_loss‚1.001507352828701 .loss/FRt2/detection_loss‚2.876150046625445 0loss/FBt1/assignment_loss‚0.2834922961588269 .loss/FRt1/detection_loss‚2.290068931579642iÕ ^ÖØ 0loss/FRt4/assignment_loss‚3.1928676495680937 2loss/SRqqt3/assignment_loss‚0.9920226504643247 .loss/FRt4/detection_loss‚2.766333725001361 0loss/SRqqt3/detection_loss‚1.001507352828701 .loss/FRt2/detection_loss‚2.876150046625445 0loss/FBt1/assignment_loss‚0.2834922961588269 .loss/FRt1/detection_loss‚2.290068931579642 /loss/FRt3/detection_loss‚1.3954136008281766 1loss/SRqqt1/detection_loss‚1.9854332864284516 % _timestamp‚1.7885327770387433e+09 0loss/FRt3/assignment_loss‚1.4214408611073905 /loss/FBt2/detection_loss‚2.4321776470717262 /loss/FBt4/detection_loss‚1.1996294883164493 /loss/FRt1/assignment_loss‚4.809950606822968 2loss/SRqqt2/assignment_loss‚1.8562748270301983 0loss/SRqqt4/detection_loss‚2.969901767373085 %loss/total_loss‚45.22396721881279 epoch‚2 1loss/FBt3/assignment_loss‚0.02757631320520503 1loss/FBt4/assignment_loss‚0.24495806341399823 .loss/FBt1/detection_loss‚2.145949512720108 /loss/FBt3/detection_loss‚0.9610073503695036 _runtime‚3761.192181044 2loss/SRqqt1/assignment_loss‚1.5900925731286406 2loss/SRqqt4/assignment_loss‚1.9627522885799409 0loss/FBt2/assignment_loss‚0.3065152351011741 1loss/SRqqt2/detection_loss‚2.6653938910056807 trainer/global_step‚3049 /loss/FRt2/assignment_loss‚3.847057254083695 _step‚249ù Ï¥;ׂ(:… ¬°ëÔŒ”˜ gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5748 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U¿¡`~Ø‚(:ó ¬°ëÔÜùëš disk.dm-1.out‚ 119.98828125 disk.drbd1260.in‚0 disk.sda.out‚ 119.98828125 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚73691.9482421875' disk.nvme0c0n1.out‚73691.9482421875 disk.sda.in‚0! disk.nvme0n1.in‚ 8738.93359375 disk.sda3.in‚0 disk.sda3.out‚ 119.98828125# memory_percent‚92.5985336303711# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 network.recv‚ 1.289933e+06) disk./.usagePercent‚78.91492891141829 disk.dm-0.in‚ 8738.93359375 disk.dm-1.in‚0 cpu‚25.305755956398073% disk./.usageGB‚1175.8894691467285# disk.nvme0c0n1.in‚ 8738.93359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.51171875& proc.memory.availableMB‚ 1212.65625 proc.cpu.threads‚17 network.sent‚ 6.902523e+06" disk.dm-0.out‚73691.9482421875ÿ‘…F‘Ù‚(:† »°ëÔüËË– gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚104.009' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8018 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y(£rÚ‚(:ç »°ëÔúÿ÷— disk.dm-1.in‚0 disk.sda.out‚ 120.4453125$ disk./.usageGB‚1175.980136871338! disk.nvme0n1.in‚ 8738.93359375" disk.nvme0n1.out‚ 74101.8828125# disk.nvme0c0n1.in‚ 8738.93359375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.515625$ memory_percent‚92.60013103485107# proc.memory.rssMB‚ 3043.94140625 disk.sda3.out‚ 120.4453125 disk.dm-0.out‚ 74101.8828125 disk.dm-1.out‚ 120.4453125) proc.memory.availableMB‚ 1212.39453125* proc.memory.percent‚18.578743934631348 cpu‚25.304922198305306 proc.cpu.threads‚17$ disk.nvme0c0n1.out‚ 74101.8828125 network.sent‚ 7.010992e+06 network.recv‚ 1.296126e+06( disk./.usagePercent‚78.9210136984921 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 8738.93359375rqë‘Û‚(:† Ê°ëÔÕýº– gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.678' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3356 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ÿm˜nÜ‚(:ã Ê°ëÔºÒÀš disk.dm-0.out‚ 74327.15625 disk.sda2.in‚0) proc.memory.availableMB‚ 1212.88671875 network.sent‚ 7.01593e+06 disk.nvme0n1.out‚ 74327.15625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.515625) disk./.usagePercent‚78.92551866307213$ disk./.usageGB‚1176.047264099121# memory_percent‚92.5971269607544# proc.memory.rssMB‚ 3043.94140625 cpu‚25.303954497191395 proc.cpu.threads‚17 network.recv‚ 1.299732e+06" disk.nvme0c0n1.out‚ 74327.15625 disk.dm-0.in‚ 8738.93359375 disk.dm-1.out‚ 120.91015625 disk.sda.in‚0 disk.sda.out‚ 120.91015625 disk.sda2.out‚0* proc.memory.percent‚18.578743934631348! disk.nvme0n1.in‚ 8738.93359375 disk.sda3.out‚ 120.91015625# disk.nvme0c0n1.in‚ 8738.93359375.ê…¢‘Ý‚(:† Ù°ëÔñ‰å— gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6088 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡x’ƒ~Þ‚(:ó Ù°ëÔÈŸÁœ disk.dm-0.in‚ 8762.30078125$ memory_percent‚92.59622097015381 disk.dm-1.out‚ 121.31640625 disk.sda2.in‚0$ disk.nvme0n1.out‚74523.525390625 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 8762.30078125* proc.memory.percent‚18.578743934631348 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.51953125! disk.nvme0n1.in‚ 8762.30078125 disk.sda3.out‚ 121.31640625# proc.memory.rssMB‚ 3043.94140625 cpu‚25.303196946034458 proc.cpu.threads‚17) disk./.usagePercent‚78.92625954887622 disk.sda.in‚0 disk.sda.out‚ 121.31640625& disk.nvme0c0n1.out‚74523.525390625! disk.dm-0.out‚74523.525390625) proc.memory.availableMB‚ 1213.03515625 network.sent‚ 7.02246e+06 network.recv‚ 1.304374e+06% disk./.usageGB‚1176.0583038330078 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0™üpo2ß- _wandbruntime‚3817  _runtime‚38171ØâÆàÀ _runtime‚ 3817.30201441 _step‚250 & lr-AdamW/pg1‚0.0007573136625621187 & lr-AdamW/pg2‚0.0007573136625621187 trainer/global_step‚3099 % _timestamp‚1.7885328331484828e+09*ìYËáÅ _step‚250 & lr-AdamW/pg1‚0.0007573136625621187 & lr-AdamW/pg2‚0.0007573136625621187 trainer/global_step‚3099 % _timestamp‚1.7885328331484828e+09 _runtime‚ 3817.30201441úÝjc2â- _wandbruntime‚3817  _runtime‚3817â6BÜ'ãÖO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84093467 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139317685901824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131740765229568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125592741438464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133063011503616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119578229849232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131741016900608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885328331496215e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84067124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22252739 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84067124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚3817.303338435 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6130620324455 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119578212809872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139317377633280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57220253 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57219589 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125592723435520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6148275462144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6148275462144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6148041794048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142696590 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125592723435520 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84067124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚3817.303338435 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6130620324455 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119578212809872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139317377633280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57220253 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57219589 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125592723435520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6148275462144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6148275462144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6148041794048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142696590 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125592723435520 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84093467 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139317685901824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131740765229568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125592741438464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133063011503616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119578229849232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131741016900608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885328331496215e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84067124 TGDeviceStatsMonitor.on_train_batch_start/inactive_splH,ýÄôit.small_pool.freed‚22252739 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0û–ÆÇ•2å- _wandbruntime‚3818  _runtime‚3818AÒ "å&æßM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84120185 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84093816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119615743167296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131782266976768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142741583 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚252 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84120185 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133104671533568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚3099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58621396 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139361063098368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125632009472512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142715212 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58621398 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131782015292416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22260632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133104695473664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34977652 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125748568001385 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6132578813441 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34977654 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142715212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84093816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22259971 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6256671685632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6150239501312 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125632027475456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6150239501312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6150005819904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58621398 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131782266976768 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125632009472512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125748321980737 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6132807794729 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142741583 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57237623 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58621396 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119615760206656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6150005819904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120832 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582à ÝZê&çäM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6132807794729 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142741583 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57237623 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58621396 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119615760206656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6150005819904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120832 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84120185 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84093816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119615743167296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131782266976768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142741583 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚252 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84120185 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133104671533568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚3099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58621396 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139361063098368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125632009472512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142715212 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58621398 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131782015292416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22260632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133104695473664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34977652 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125748568001385 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6132578813441 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34977654 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142715212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84093816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22259971 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6256671685632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6150239501312 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125632027475456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6150239501312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6150005819904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58621398 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131782266976768 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125632009472512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125748321980737ü2‹ë2è- _wandbruntime‚3818  _runtime‚3818õ=ÓŒ]é× 2loss/SRqqt1/assignment_loss‚1.3712371883815841 0loss/FBt4/assignment_loss‚0.2338083099649713 .loss/FBt1/detection_loss‚2.506134933233261 _runtime‚3818.296069829 0loss/FRt3/assignment_loss‚1.1466989253054967 .loss/FRt1/detection_loss‚2.415003389120102 trainer/global_step‚3099 epoch‚2 % _timestamp‚1.7885328341427093e+09 /loss/FRt4/assignment_loss‚3.916441172361374 1loss/FBt1/assignment_loss‚0.23309910474861795 1loss/SRqqt3/detection_loss‚0.8843839173870427 .loss/FBt2/detection_loss‚2.911912108461062 2loss/SRqqt2/assignment_loss‚1.7398403029356684 4loss/FBt3/assignment_loss‚0.00021267868928059424 /loss/FRt4/detection_loss‚2.7019607209497027 _step‚253 /loss/FRt1/assignment_loss‚4.062344504030127 2loss/SRqqt3/assignment_loss‚0.9610360526080642 .loss/FRt3/detection_loss‚1.425673012833365 0loss/SRqqt1/detection_loss‚2.080704616088616 1loss/SRqqt4/assignment_loss‚2.003376496334871 1loss/FBt2/assignment_loss‚0.23897742234718558 /loss/FRt2/detection_loss‚2.8348621241748333 1loss/SRqqt2/detection_loss‚2.4361343145370484 1loss/SRqqt4/detection_loss‚2.5469216157992682 /loss/FRt2/assignment_loss‚4.086848214268684 /loss/FBt3/detection_loss‚0.9330806940592624 /loss/FBt4/detection_loss‚1.5016978214825354 &loss/total_loss‚45.172389640102026táܳbêÜ 1loss/FBt2/assignment_loss‚0.23897742234718558 /loss/FRt2/detection_loss‚2.8348621241748333 1loss/SRqqt2/detection_loss‚2.4361343145370484 1loss/SRqqt4/detection_loss‚2.5469216157992682 /loss/FRt2/assignment_loss‚4.086848214268684 /loss/FBt3/detection_loss‚0.9330806940592624 /loss/FBt4/detection_loss‚1.5016978214825354 &loss/total_loss‚45.172389640102026 2loss/SRqqt1/assignment_loss‚1.3712371883815841 0loss/FBt4/assignment_loss‚0.2338083099649713 .loss/FBt1/detection_loss‚2.506134933233261 _runtime‚3818.296069829 0loss/FRt3/assignment_loss‚1.1466989253054967 .loss/FRt1/detection_loss‚2.415003389120102 trainer/global_step‚3099 epoch‚2 % _timestamp‚1.7885328341427093e+09 /loss/FRt4/assignment_loss‚3.916441172361374 1loss/FBt1/assignment_loss‚0.23309910474861795 1loss/SRqqt3/detection_loss‚0.8843839173870427 .loss/FBt2/detection_loss‚2.911912108461062 2loss/SRqqt2/assignment_loss‚1.7398403029356684 4loss/FBt3/assignment_loss‚0.00021267868928059424 /loss/FRt4/detection_loss‚2.7019607209497027 _step‚253 /loss/FRt1/assignment_loss‚4.062344504030127 2loss/SRqqt3/assignment_loss‚0.9610360526080642 .loss/FRt3/detection_loss‚1.425673012833365 0loss/SRqqt1/detection_loss‚2.080704616088616 1loss/SRqqt4/assignment_loss‚2.003376496334871ýDEœë‚(:‘ è°ëÔ“­– gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.428' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.685599999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AÛ\ê}ì‚(:ò è°ëÔ¹õΖ disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚ 7.02717e+06 disk.sda3.out‚ 121.72265625 disk.dm-0.in‚ 8879.50390625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 disk.sda.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚74620.025390625# disk.nvme0c0n1.in‚ 8879.50390625! disk.dm-0.out‚74620.025390625 disk.sda1.out‚0) disk./.usagePercent‚78.92889284237341 disk.sda.out‚ 121.72265625& disk.nvme0c0n1.out‚74620.025390625 disk.dm-1.out‚ 121.72265625 disk.drbd1260.out‚ 0.51953125$ memory_percent‚92.59355068206787) proc.memory.availableMB‚ 1213.47265625 cpu‚25.302422378156425 network.recv‚ 1.307491e+06$ disk./.usageGB‚1176.097541809082 disk.sda2.in‚0! disk.nvme0n1.in‚ 8879.50390625!¤›í‚(: ÷°ëÔ¹Þ—• gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.712799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È…­¤‚î‚(:÷ ÷°ëÔ’¿©› disk.sda1.out‚0 disk.drbd1260.out‚ 0.52734375# disk.nvme0c0n1.in‚ 8879.50390625( disk.nvme0c0n1.out‚74998.54833984375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 8879.50390625 disk.dm-0.in‚ 8879.50390625# disk.dm-0.out‚74998.54833984375$ memory_percent‚92.59977340698242# proc.memory.rssMB‚ 3043.94140625 network.sent‚ 7.136018e+06 disk.sda3.in‚0 disk.sda3.out‚ 122.28515625' proc.memory.availableMB‚ 1212.453125 cpu‚25.301704211118615% disk./.usageGB‚1176.1597938537598 disk.sda.out‚ 122.28515625 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚74998.54833984375* proc.memory.percent‚18.578743934631348 proc.cpu.threads‚17 network.recv‚ 1.31455e+06) disk./.usagePercent‚78.93307062932469 disk.dm-1.in‚0 disk.dm-1.out‚ 122.28515625ŸÌ› ‘ï‚(:† †±ëÔ„©Œ– gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8972 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ‘žCƒð‚(:ø †±ëÔ츈› cpu‚25.30086651819721 disk.sda.in‚0 disk.sda.out‚ 122.62109375 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 8879.50390625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚75360.19287109375# disk.dm-0.out‚75360.19287109375$ memory_percent‚92.59541034698486# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 network.sent‚ 7.140977e+06 network.recv‚ 1.317628e+06) disk./.usagePercent‚78.94987752394755 disk.sda1.in‚0! disk.nvme0n1.in‚ 8879.50390625 disk.sda3.in‚0( disk.nvme0c0n1.out‚75360.19287109375 disk.dm-0.in‚ 8879.50390625$ disk./.usageGB‚1176.410228729248 disk.drbd1260.out‚ 0.52734375 disk.sda3.out‚ 122.62109375 disk.dm-1.in‚0 disk.dm-1.out‚ 122.62109375) proc.memory.availableMB‚ 1213.16796875ÏÊ9„‘ñ‚(:† •±ëÔ¤Îâ• gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚139.769' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9538 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FM’uò‚(:ê •±ëÔñ§í— disk.sda3.in‚0( proc.memory.availableMB‚ 1212.9140625# proc.memory.rssMB‚ 3043.94140625 disk.sda2.in‚0$ memory_percent‚92.59696006774902* proc.memory.percent‚18.578743934631348 cpu‚25.300137555635096 proc.cpu.threads‚17 network.sent‚ 7.147318e+06 network.recv‚ 1.322464e+06) disk./.usagePercent‚78.95141817728862# disk.nvme0c0n1.in‚ 8879.50390625 disk.dm-0.out‚ 75709.1015625 disk.dm-1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 8879.50390625% disk./.usageGB‚1176.4331855773926$ disk.nvme0c0n1.out‚ 75709.1015625 disk.dm-0.in‚ 8879.50390625 disk.dm-1.out‚ 122.98828125 disk.drbd1260.in‚0 disk.drbd1260.out‚0.53125" disk.nvme0n1.out‚ 75709.1015625 disk.sda3.out‚ 122.98828125 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 122.98828125 disk.sda2.out‚0³2ó- _wandbruntime‚3873  _runtime‚38732hÆôÀ _step‚254 & lr-AdamW/pg1‚0.0007542182035906446 & lr-AdamW/pg2‚0.0007542182035906446 trainer/global_step‚3149 % _timestamp‚1.7885328893476872e+09 _runtime‚ 3873.50137458ÇvAZËõÅ trainer/global_step‚3149 % _timestamp‚1.7885328893476872e+09 _runtime‚ 3873.50137458 _step‚254 & lr-AdamW/pg1‚0.0007542182035906446 & lr-AdamW/pg2‚0.0007542182035906446þ/&B2ö- _wandbruntime‚3873  _runtime‚3873~dI–I÷ÖO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127683535066475 C;DeviceStatsMonitor.on_train_batch_start/segment.all.curŽ\<“rent‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127557043288064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127557025285120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6354672884736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6246491168768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133803534456832 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85405076 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58107094 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135147186065920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3873.503530603 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144921947 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127557043288064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144948292 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35506952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133803282785792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59516873 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6228558364963 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133803282785792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6246257500672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6246257500672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59516871 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127683289046035 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141501564136448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885328893494601e+09 _step‚255 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85431419 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6246491168768 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127683535066475 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127557043288064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127557025285120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6354672884736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6246491168768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133803534456832 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85405076 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58107094 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135147186065920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3873.503530603 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144921947 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127557043288064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144948292 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35506952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133803282785792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59516873 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6228558364963 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133803282785792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6246257500672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6246257500672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59516871 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127683289046035 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141501564136448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885328893494601e+09 _step‚255 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÿú‹†2ù- _wandbruntime‚3874  _runtime‚3874ëÝOä&úÞM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85458137 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127596329325056 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127722777892445 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133844784532992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133844532848640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59535148 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59535148 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35517754 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144993285 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144966914 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6356397239808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121492261038496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6230745835237 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6356679457792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚256 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6248455207936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58123940 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 _runtime‚3874.547509353 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127596311322112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6248221526528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133844784532992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141545573433856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144993285 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135188893976064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127723023913093 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135188870035968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6248455207936 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6248221526528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2²çšŒSûãM _step‚256 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6248455207936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58123940 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 _runtime‚3874.547509353 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127596311322112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6248221526528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133844784532992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141545573433856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144993285 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135188893976064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127723023913093 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135188870035968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6248455207936 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6248221526528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85458137 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127596329325056 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127722777892445 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133844784532992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133844532848640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59535148 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59535148 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35517754 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144993285 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144966914 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6356397239808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121492261038496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6230745835237 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6356679457792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0€f‹ Y2ü- _wandbruntime‚3874  _runtime‚38742éàZýÔ &loss/total_loss‚44.988231400233815 /loss/FRt1/assignment_loss‚4.330854258886197 1loss/FBt4/assignment_loss‚0.20123666640808863 _step‚257 /loss/FRt2/detection_loss‚2.8594574268375124 /loss/FRt2/assignment_loss‚3.093468734196254 /loss/FBt3/detection_loss‚0.9169515331089496 0loss/SRqqt3/detection_loss‚1.086451508882253 % _timestamp‚1.7885328903959308e+09 trainer/global_step‚3149 /loss/FRt3/assignment_loss‚2.155532317608595 2loss/SRqqt2/assignment_loss‚1.8027753705779712 2loss/SRqqt3/assignment_loss‚1.0309115632396677 .loss/FRt1/detection_loss‚2.352599808355659 /loss/FRt4/detection_loss‚2.4849065211084156 /loss/FBt1/detection_loss‚2.5985849076195766 1loss/SRqqt4/assignment_loss‚2.236722176113436 0loss/FBt2/assignment_loss‚0.6583043390133831 1loss/SRqqt1/detection_loss‚2.0848167715845882 1loss/SRqqt2/detection_loss‚2.7073980569839478 /loss/FBt4/detection_loss‚1.2519396470652686 epoch‚2 _runtime‚3874.549269781 1loss/SRqqt1/assignment_loss‚1.720486790545889 0loss/FBt1/assignment_loss‚0.3103438584492228 0loss/SRqqt4/detection_loss‚2.063427852046105 0loss/FRt4/assignment_loss‚3.2784962438874774 /loss/FRt3/detection_loss‚1.5544513140805065 1loss/FBt3/assignment_loss‚0.06748758559755515 /loss/FBt2/detection_loss‚2.1406261480372883Úø._þÙ 1loss/SRqqt1/assignment_loss‚1.720486790545889 0loss/FBt1/assignment_loss‚0.3103438584492228 0loss/SRqqt4/detection_loss‚2.063427852046105 0loss/FRt4/assignment_loss‚3.2784962438874774 /loss/FRt3/detection_loss‚1.5544513140805065 1loss/FBt3/assignment_loss‚0.06748758559755515 /loss/FBt2/detection_loss‚2.1406261480372883 &loss/total_loss‚44.988231400233815 /loss/FRt1/assignment_loss‚4.330854258886197 1loss/FBt4/assignment_loss‚0.20123666640808863 _step‚257 /loss/FRt2/detection_loss‚2.8594574268375124 /loss/FRt2/assignment_loss‚3.093468734196254 /loss/FBt3/detection_loss‚0.9169515331089496 0loss/SRqqt3/detection_loss‚1.086451508882253 % _timestamp‚1.7885328903959308e+09 trainer/global_step‚3149 /loss/FRt3/assignment_loss‚2.155532317608595 2loss/SRqqt2/assignment_loss‚1.8027753705779712 2loss/SRqqt3/assignment_loss‚1.0309115632396677 .loss/FRt1/detection_loss‚2.352599808355659 /loss/FRt4/detection_loss‚2.4849065211084156 /loss/FBt1/detection_loss‚2.5985849076195766 1loss/SRqqt4/assignment_loss‚2.236722176113436 0loss/FBt2/assignment_loss‚0.6583043390133831 1loss/SRqqt1/detection_loss‚2.0848167715845882 1loss/SRqqt2/detection_loss‚2.7073980569839478 /loss/FBt4/detection_loss‚1.2519396470652686 epoch‚2 _runtime‚3874.549269781ýFëÿ‚(:… ¤±ëÔð”Ö™ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2168 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼Z®ke€‚(:Ú ¤±ëÔâîë› disk.sda.in‚0 disk.sda.out‚123.375 disk.nvme0n1.in‚ 9002.328125$ disk.nvme0n1.out‚76022.869140625 disk.sda3.out‚123.375 disk.dm-1.out‚123.375 proc.cpu.threads‚17 network.recv‚ 1.325875e+06( disk./.usagePercent‚78.9766193029517 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 9002.328125& disk.nvme0c0n1.out‚76022.869140625! disk.dm-0.out‚76022.869140625 network.sent‚ 7.152261e+06 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0) proc.memory.availableMB‚ 1213.08203125# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348% disk./.usageGB‚1176.8087005615234 disk.sda1.out‚0 disk.drbd1260.out‚0.53125 disk.dm-0.in‚ 9002.328125$ memory_percent‚92.59593486785889 cpu‚25.29938230193797…‚(:„ ѱëÔš gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.925 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f-«òw†‚(:ì ѱëÔ¨‡î  disk.drbd1260.out‚ 0.5390625 disk.sda3.in‚0 disk.sda3.out‚ 124.63671875 proc.cpu.threads‚17 network.sent‚ 7.27109e+06) disk./.usagePercent‚78.99330408198074$ disk./.usageGB‚1177.057315826416 disk.dm-0.in‚ 9002.328125 disk.dm-1.out‚ 124.63671875 disk.sda.in‚0 disk.sda.out‚ 124.63671875 disk.sda2.in‚0 network.recv‚ 1.33939e+06 disk.sda2.out‚0 disk.nvme0n1.in‚ 9002.328125% disk.nvme0n1.out‚76867.2685546875# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 cpu‚25.297180436698763! disk.nvme0c0n1.in‚ 9002.328125' disk.nvme0c0n1.out‚76867.2685546875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚92.59772300720215( proc.memory.availableMB‚ 1212.7890625" disk.dm-0.out‚76867.2685546875‘òÉ`2‡- _wandbruntime‚3933  _runtime‚3933ïøĈ¾ % lr-AdamW/pg1‚0.000751050656962896 % lr-AdamW/pg2‚0.000751050656962896 trainer/global_step‚3199 $ _timestamp‚1.788532948875994e+09 _runtime‚3933.029556363 _step‚258±_w"ɉà _runtime‚3933.029556363 _step‚258 % lr-AdamW/pg1‚0.000751050656962896 % lr-AdamW/pg2‚0.000751050656962896 trainer/global_step‚3199 $ _timestamp‚1.788532948875994e+09‚…2Š- _wandbruntime‚3933  _runtime‚3933ê×Å#Ü'‹ÖO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6326496405471 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6344706875392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137231408508416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129657744957743 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60430621 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129521345137664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129521327134720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137231384568320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58994278 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚3933.030967463 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326725386551 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135866052013056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123331248552272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147199994 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86743028 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135865800342016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36047050 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36047052 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6344473207296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60430623 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129521327134720 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123331265591632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚259 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86769371 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6344473207296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60430621 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86743028 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22946529 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚3199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143685706468352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60430623 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86769371 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752e–iá'ŒÛO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147199994 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86743028 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135865800342016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36047050 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36047052 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6344473207296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60430623 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129521327134720 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123331265591632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚259 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86769371 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6344473207296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60430621 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86743028 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22946529 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚3199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143685706468352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60430623 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86769371 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6326496405471 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6344706875392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137231408508416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129657744957743 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60430621 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129521345137664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129521327134720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137231384568320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58994278 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚3933.030967463 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326725386551 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135866052013056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123331248552272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2ƒØôÏ{2- _wandbruntime‚3934  _runtime‚3934Ž>†•æ&ŽàM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143729377970176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123368795949056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6328683875745 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60448896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59011689 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147244987 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6346670914560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6346437233152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86769720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129560631174656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7885329499721396e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86796089 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129697233804153 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135907302089216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60448898 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36057852 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22953837 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137273068538368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6456309431808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137273092478464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129697479824801 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135907050404864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60448898 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚3934.126234734 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147218616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143729698808320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123368778909696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6346437233152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60448896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86796089 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129560613171712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36057854 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147244987 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86769720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129560631174656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7885329499721396e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86796089 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129697233804153 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135907302089216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60448898 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36057852 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22953837 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137273068538368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6456309431808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137273092478464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129697479824801 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135907050404864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60448898 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚3934.126234734 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147218616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143729698808320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123368778909696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6346437233152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60448896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86796089 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129560613171712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36057854 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147244987 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59011689 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147244987 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6346670914560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6346437233152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280„'Ü<2- _wandbruntime‚3934  _runtime‚3934½i\‘Ö 1loss/FBt1/assignment_loss‚0.35526106811322766 /loss/FRt4/detection_loss‚2.8074652860904563 1loss/SRqqt4/detection_loss‚2.5252267146110534 /loss/FBt1/detection_loss‚2.6670223887150106 %loss/total_loss‚44.23758268768792 /loss/FBt3/detection_loss‚0.9788438539641598 /loss/FRt1/assignment_loss‚3.285439470718647 0loss/SRqqt1/assignment_loss‚1.45611199609151 /loss/FRt1/detection_loss‚2.7457771753442697 1loss/SRqqt3/detection_loss‚1.0107377156545625 0loss/FRt3/assignment_loss‚1.5204647764563561 2loss/SRqqt2/assignment_loss‚1.9857615649700164 1loss/FBt2/assignment_loss‚0.25526198843861336 /loss/FRt3/detection_loss‚1.5622111530276015 /loss/FBt2/detection_loss‚2.2728316680244776 .loss/FBt4/detection_loss‚1.143068887095185 epoch‚2 /loss/FRt2/assignment_loss‚3.518587863003766 $ _timestamp‚1.788532949975077e+09 trainer/global_step‚3199 0loss/FRt4/assignment_loss‚3.0191206459341378 2loss/SRqqt4/assignment_loss‚1.8645839965343476 3loss/FBt3/assignment_loss‚8.633768808863351e-05 0loss/SRqqt2/detection_loss‚2.761806461215019 _step‚261 _runtime‚3934.128480085 /loss/FRt2/detection_loss‚3.0758617696938693 2loss/SRqqt3/assignment_loss‚1.1551738725017722 0loss/FBt4/assignment_loss‚0.2231777618997928 0loss/SRqqt1/detection_loss‚2.047698271901984ºDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61344371 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131485628984320 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59881912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131632200869451 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36587150 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6442922582016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139315607010816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61344373 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125207783462832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137928569569280 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23294076 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6554512064000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚263 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137928317898240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88107323 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139315583070720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6554227735552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6442922582016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88080980 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149451696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61344371 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6424663427059 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚3249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61344373 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6424434445979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125207766423472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944©®èá' ÛO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6442922582016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88080980 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149451696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61344371 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6424663427059 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚3249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61344373 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6424434445979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125207766423472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145870119074816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61344371 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131485628984320 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59881912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131632200869451 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36587150 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6442922582016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139315607010816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61344373 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125207783462832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137928569569280 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23294076 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6554512064000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚263 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137928317898240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88107323 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139315583070720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6554227735552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532‡ãUì2¡- _wandbruntime‚3991  _runtime‚3991ä\pUæ&¢àM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125245313820256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61362646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36597952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6444652939776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88107672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149496689 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145913489360384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6556222319616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6426621916253 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6426392934965 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61362648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59898743 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149496689 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137969567961088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125245296780896 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88134041 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131524915021312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61362646 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23301474 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139357267040768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6444652939776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88107672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59899428 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139357290980864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 _runtime‚3991.401393918 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131524933024256 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6444886621184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131524915021312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120Y»'³£åM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149496689 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137969567961088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125245296780896 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88134041 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131524915021312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61362646 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23301474 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139357267040768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6444652939776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88107672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59899428 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139357290980864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 _runtime‚3991.401393918 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131524933024256 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6444886621184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131524915021312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125245313820256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61362646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36597952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6444652939776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88107672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149496689 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145913489360384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6556222319616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6426621916253 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6426392934965 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61362648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59898743 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ˆÌGß2¤- _wandbruntime‚3991  _runtime‚3991Šc¼ˆ]¥× 1loss/SRqqt3/detection_loss‚0.9432634315541331 %loss/total_loss‚44.61264769509966 2loss/SRqqt1/assignment_loss‚1.4847560363800028 1loss/SRqqt2/assignment_loss‚2.101020000874996 2loss/SRqqt4/assignment_loss‚1.9978425882079385 /loss/FRt1/detection_loss‚2.5434110646738723 .loss/FRt2/detection_loss‚3.723496988415718 /loss/FBt1/detection_loss‚2.5893175389085497 2loss/SRqqt3/assignment_loss‚0.9210673252577489 0loss/FBt2/assignment_loss‚0.4598478019146796 /loss/FBt2/detection_loss‚2.2400302648544312 trainer/global_step‚3249 _step‚265 .loss/FRt4/detection_loss‚2.946696476294444 /loss/FBt4/detection_loss‚1.3653845225389187 3loss/FBt3/assignment_loss‚4.999277747774755e-06 /loss/FRt3/detection_loss‚1.5047864096579693 epoch‚2 % _timestamp‚1.7885330072490997e+09 _runtime‚3991.402556911 0loss/FRt2/assignment_loss‚3.9791488721966743 0loss/FBt4/assignment_loss‚0.2160000819023352 1loss/SRqqt1/detection_loss‚1.8181432298449582 /loss/FBt3/detection_loss‚0.6100038494332694 /loss/FRt4/assignment_loss‚3.040617814430824 1loss/SRqqt4/detection_loss‚2.4303619130091234 /loss/FRt1/assignment_loss‚3.508660484762753 0loss/FRt3/assignment_loss‚1.4907024387390382 0loss/FBt1/assignment_loss‚0.2819046856156839 1loss/SRqqt2/detection_loss‚2.4161788763538485ä¾Èb¦Ü /loss/FRt1/assignment_loss‚3.508660484762753 0loss/FRt3/assignment_loss‚1.4907024387390382 0loss/FBt1/assignment_loss‚0.2819046856156839 1loss/SRqqt2/detection_loss‚2.4161788763538485 1loss/SRqqt3/detection_loss‚0.9432634315541331 %loss/total_loss‚44.61264769509966 2loss/SRqqt1/assignment_loss‚1.4847560363800028 1loss/SRqqt2/assignment_loss‚2.101020000874996 2loss/SRqqt4/assignment_loss‚1.9978425882079385 /loss/FRt1/detection_loss‚2.5434110646738723 .loss/FRt2/detection_loss‚3.723496988415718 /loss/FBt1/detection_loss‚2.5893175389085497 2loss/SRqqt3/assignment_loss‚0.9210673252577489 0loss/FBt2/assignment_loss‚0.4598478019146796 /loss/FBt2/detection_loss‚2.2400302648544312 trainer/global_step‚3249 _step‚265 .loss/FRt4/detection_loss‚2.946696476294444 /loss/FBt4/detection_loss‚1.3653845225389187 3loss/FBt3/assignment_loss‚4.999277747774755e-06 /loss/FRt3/detection_loss‚1.5047864096579693 epoch‚2 % _timestamp‚1.7885330072490997e+09 _runtime‚3991.402556911 0loss/FRt2/assignment_loss‚3.9791488721966743 0loss/FBt4/assignment_loss‚0.2160000819023352 1loss/SRqqt1/detection_loss‚1.8181432298449582 /loss/FBt3/detection_loss‚0.6100038494332694 /loss/FRt4/assignment_loss‚3.040617814430824 1loss/SRqqt4/detection_loss‚2.4303619130091234‰njøp§‚(:å œ²ëÔ„惛& disk.nvme0c0n1.out‚78396.814453125! disk.dm-0.out‚78396.814453125 disk.dm-1.out‚ 126.78125 disk.sda1.out‚0 disk.drbd1260.in‚0) disk./.usagePercent‚79.02689815864635% disk./.usageGB‚1177.5578918457031 disk.sda3.out‚ 126.78125 disk.dm-0.in‚ 9284.7734375 disk.sda.out‚ 126.78125 disk.nvme0n1.in‚ 9284.7734375$ memory_percent‚92.59717464447021# proc.memory.rssMB‚ 3043.94140625" disk.nvme0c0n1.in‚ 9284.7734375 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚78396.814453125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.out‚0.546875) proc.memory.availableMB‚ 1212.87890625* proc.memory.percent‚18.578743934631348 cpu‚25.293657692702187 network.sent‚ 7.414013e+06 network.recv‚ 1.36097e+06 disk.sda3.in‚0;N±¨‚(:… œ²ëÔß͸¨ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.645' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.929 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 þóÌ©‚(:… «²ëÔÒñ…– gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.649 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;}¡Hoª‚(:ä «²ëÔá•— network.recv‚ 1.367257e+06% disk./.usageGB‚1177.3702774047852! disk.dm-0.out‚78628.947265625 disk.sda1.in‚0$ disk.nvme0n1.out‚78628.947265625 disk.sda3.out‚127.1875 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.55078125 network.sent‚ 7.509679e+06) disk./.usagePercent‚79.01430719609735" disk.nvme0c0n1.in‚ 9284.7734375& disk.nvme0c0n1.out‚78628.947265625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚127.1875 disk.nvme0n1.in‚ 9284.7734375 disk.sda3.in‚0$ memory_percent‚92.60029792785645( proc.memory.availableMB‚ 1212.3671875# proc.memory.rssMB‚ 3043.94140625* proc.memory.percent‚18.578743934631348 disk.dm-0.in‚ 9284.7734375 disk.dm-1.out‚127.1875 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.292930112777558ä´À«‚(:„ º²ëÔÂÞà• gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.884 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ:˜jy¬‚(:î º²ëÔñë›™* proc.memory.percent‚18.578743934631348 cpu‚25.292254345628304 disk.dm-0.in‚ 9284.7734375 network.sent‚ 7.514503e+06 network.recv‚ 1.370532e+06) disk./.usagePercent‚78.99087021835271 disk.sda3.in‚0 disk.sda3.out‚ 127.59765625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.55078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 127.59765625 proc.cpu.threads‚17! disk.dm-0.out‚79135.818359375% disk./.usageGB‚1177.0210494995117" disk.nvme0c0n1.in‚ 9284.7734375& disk.nvme0c0n1.out‚79135.818359375 disk.dm-1.out‚ 127.59765625 disk.nvme0n1.in‚ 9284.7734375$ disk.nvme0n1.out‚79135.818359375 disk.drbd1260.in‚0$ memory_percent‚92.59891510009766& proc.memory.availableMB‚ 1212.59375# proc.memory.rssMB‚ 3043.94140625ÓoRCœ­‚(:‘ ɲëÔÚ„ä• gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.416' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.083199999999998 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pJ1au®‚(:ê ɲëÔ¢Ù°— disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚92.59729385375977' proc.memory.availableMB‚ 1212.859375( disk.nvme0c0n1.out‚79304.46630859375 disk.sda2.out‚0 disk.nvme0n1.in‚ 9284.7734375& disk.nvme0n1.out‚79304.46630859375* proc.memory.percent‚18.578743934631348 cpu‚25.29157205335625) disk./.usagePercent‚78.99087277842806" disk.nvme0c0n1.in‚ 9284.7734375 disk.dm-0.in‚ 9284.7734375 disk.dm-1.out‚ 128.15625 disk.sda.in‚0 disk.sda.out‚ 128.15625 disk.sda3.in‚0 disk.sda3.out‚ 128.15625 network.sent‚ 7.520918e+06% disk./.usageGB‚1177.0210876464844# disk.dm-0.out‚79304.46630859375 disk.drbd1260.out‚ 0.5546875 disk.sda2.in‚0# proc.memory.rssMB‚ 3043.94140625 proc.cpu.threads‚17 network.recv‚ 1.375323e+06\F3÷2¯- _wandbruntime‚4050  _runtime‚4050ÎU%Ç°Á trainer/global_step‚3299 % _timestamp‚1.7885330661426325e+09 _runtime‚4050.296155406 _step‚266 & lr-AdamW/pg1‚0.0007445019560718464 & lr-AdamW/pg2‚0.0007445019560718464ÇøÌ±Æ _runtime‚4050.296155406 _step‚266 & lr-AdamW/pg1‚0.0007445019560718464 & lr-AdamW/pg2‚0.0007445019560718464 trainer/global_step‚3299 % _timestamp‚1.7885330661426325e+09Š£nÀ2²- _wandbruntime‚4050  _runtime‚4050ø+ºÛ'³ÕO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89445275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37127252 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139990835454464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151703398 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6522372486487 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788533066143658e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60764604 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37127250 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23637354 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6654353242624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6654062622720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151677053 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139990835454464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141399805513216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141399781573120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62258121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6540904620544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133449948836864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6541138288640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚267 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62258121 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127084284294672 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133449930833920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89418932 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62258123 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133606902801599 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133449948836864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23638064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89418932 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6541138288640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139991087125504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127084301334032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6540904620544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4050.297331082 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139990835454464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141399805513216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141399781573120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62258121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6540904620544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133449948836864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6541138288640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚267 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62258121 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127084284294672 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133449930833920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89418932 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62258123 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133606902801599 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133449948836864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23638064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89418932 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6541138288640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139991087125504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127084301334032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6540904620544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4050.297331082 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151722020 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133646391648217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89445624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140032085517312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 $ _timestamp‚1.788533067262514e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127121814652096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6524559956761 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37138052 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4051.417340152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151748391 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37138054 trainer/global_step‚3299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6656053627904 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62276398 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133489216870912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141441489483264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62276396 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6656342137344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23644638 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140032337201664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62276396 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151748391 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140032337201664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6543102327808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62276398 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60782692 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148097831620608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148097519171072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133646145627569 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89445624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60782010 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141441465543168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89471993 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6543102327808 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62276396 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151748391 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140032337201664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6543102327808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62276398 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60782692 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148097831620608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148097519171072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133646145627569 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89445624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60782010 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141441465543168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89471993 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6543102327808 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151722020 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133646391648217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89445624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140032085517312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 $ _timestamp‚1.788533067262514e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127121814652096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6524559956761 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37138052 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4051.417340152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151748391 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37138054 trainer/global_step‚3299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6656053627904 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62276398 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133489216870912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141441489483264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62276396 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6656342137344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23644638 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140032337201664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582Œ P†g2¸- _wandbruntime‚4051  _runtime‚4051$iÝV¹Ð 1loss/SRqqt1/assignment_loss‚1.640258809470612 epoch‚2 /loss/FRt3/assignment_loss‚1.479369762908207 1loss/SRqqt4/assignment_loss‚2.299401641167976 /loss/FBt4/detection_loss‚1.5572210885584354 _step‚269 # _timestamp‚1.78853306726609e+09 ,loss/FBt1/detection_loss‚2.5506552335094 .loss/FBt2/detection_loss‚2.648803344139686 %loss/total_loss‚46.84557011932361 1loss/FBt4/assignment_loss‚0.16571779210057216 0loss/FRt2/assignment_loss‚3.8209305463298673 2loss/SRqqt3/assignment_loss‚1.0803119817733013 1loss/FBt2/assignment_loss‚0.30626140503059457 /loss/FRt3/detection_loss‚1.3495315718098302 2loss/SRqqt2/assignment_loss‚1.7853198494348261 2loss/FBt3/assignment_loss‚0.008643101379892926 .loss/FRt4/detection_loss‚3.066550229604428 /loss/FRt1/detection_loss‚2.3332229937825884 /loss/FRt2/detection_loss‚2.7806171217272357 /loss/FRt1/assignment_loss‚4.413557287624904 /loss/FRt4/assignment_loss‚3.924672626532041 0loss/SRqqt4/detection_loss‚2.614947654582836 /loss/FBt3/detection_loss‚1.3187831722199916 trainer/global_step‚3299 1loss/SRqqt3/detection_loss‚0.9498335572914416 1loss/SRqqt2/detection_loss‚2.5353993508550854 1loss/FBt1/assignment_loss‚0.18990997242792004 0loss/SRqqt1/detection_loss‚2.025650025061939 _runtime‚4051.419545007½ô-[ºÕ 1loss/FBt1/assignment_loss‚0.18990997242792004 0loss/SRqqt1/detection_loss‚2.025650025061939 _runtime‚4051.419545007 1loss/SRqqt1/assignment_loss‚1.640258809470612 epoch‚2 /loss/FRt3/assignment_loss‚1.479369762908207 1loss/SRqqt4/assignment_loss‚2.299401641167976 /loss/FBt4/detection_loss‚1.5572210885584354 _step‚269 # _timestamp‚1.78853306726609e+09 ,loss/FBt1/detection_loss‚2.5506552335094 .loss/FBt2/detection_loss‚2.648803344139686 %loss/total_loss‚46.84557011932361 1loss/FBt4/assignment_loss‚0.16571779210057216 0loss/FRt2/assignment_loss‚3.8209305463298673 2loss/SRqqt3/assignment_loss‚1.0803119817733013 1loss/FBt2/assignment_loss‚0.30626140503059457 /loss/FRt3/detection_loss‚1.3495315718098302 2loss/SRqqt2/assignment_loss‚1.7853198494348261 2loss/FBt3/assignment_loss‚0.008643101379892926 .loss/FRt4/detection_loss‚3.066550229604428 /loss/FRt1/detection_loss‚2.3332229937825884 /loss/FRt2/detection_loss‚2.7806171217272357 /loss/FRt1/assignment_loss‚4.413557287624904 /loss/FRt4/assignment_loss‚3.924672626532041 0loss/SRqqt4/detection_loss‚2.614947654582836 /loss/FBt3/detection_loss‚1.3187831722199916 trainer/global_step‚3299 1loss/SRqqt3/detection_loss‚0.9498335572914416 1loss/SRqqt2/detection_loss‚2.5353993508550854g]Œ3œ»‚(:‘ زëÔì“™ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p%ÿ·w¼‚(:ì زëÔµ´Ü) proc.memory.percent‚18.03419589996338 proc.cpu.threads‚17& disk.nvme0n1.out‚79395.45068359375# disk.dm-0.out‚79395.45068359375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.5546875 cpu‚25.290912066479105 disk.sda2.in‚0! disk.nvme0n1.in‚ 9408.68359375 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 9408.68359375( disk.nvme0c0n1.out‚79395.45068359375 disk.dm-1.in‚0 disk.dm-1.out‚ 128.46875 disk.sda1.in‚0 network.sent‚ 7.525746e+06) disk./.usagePercent‚78.99086945033011 disk.sda.in‚0 disk.sda.out‚ 128.46875 disk.sda2.out‚0 disk.sda3.out‚ 128.46875 disk.dm-0.in‚ 9408.68359375 disk.drbd1260.in‚0 network.recv‚ 1.378516e+06# disk./.usageGB‚1177.02103805542# memory_percent‚92.0543909072876) proc.memory.availableMB‚ 1301.80859375# proc.memory.rssMB‚ 2954.722656254£‘½‚(:† ç²ëÔïÏÊ– gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7794 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ!¤(¾‚(:ö ç²ëÔ·ð— disk.dm-1.out‚ 128.87109375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2954.72265625 disk.sda.out‚ 128.87109375 disk.sda3.out‚ 128.87109375$ memory_percent‚92.05451011657715( proc.memory.availableMB‚ 1301.7890625 proc.cpu.threads‚17( disk./.usagePercent‚78.9856745454453# disk.dm-0.out‚79761.82958984375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd1260.out‚ 0.55859375 disk.sda2.in‚0 disk.sda3.in‚0) proc.memory.percent‚18.03419589996338 network.sent‚ 7.634319e+06 network.recv‚ 1.385078e+06# disk.nvme0c0n1.in‚ 9408.68359375 disk.dm-0.in‚ 9408.68359375 disk.sda.in‚0! disk.nvme0n1.in‚ 9408.68359375& disk.nvme0n1.out‚79761.82958984375 cpu‚25.29016976324857% disk./.usageGB‚1176.9436302185059( disk.nvme0c0n1.out‚79761.82958984375Fl~šœ¿‚(:‘ ö²ëԨģ• gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.379199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐþU.€À‚(:õ ö²ëÔÄÕ‡™ network.sent‚ 7.639406e+06$ disk./.usageGB‚1177.078224182129 disk.dm-0.in‚ 9408.68359375 disk.dm-1.out‚ 129.74609375 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 129.74609375 disk.sda2.in‚0 network.recv‚ 1.388489e+06) disk./.usagePercent‚78.99470725927591 disk.sda3.out‚ 129.74609375) proc.memory.percent‚18.03419589996338 cpu‚25.28946417406803( disk.nvme0c0n1.out‚79945.57177734375# disk.dm-0.out‚79945.57177734375 disk.dm-1.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 9408.68359375 disk.sda3.in‚0# memory_percent‚92.0529842376709( proc.memory.availableMB‚ 1302.0390625# disk.nvme0c0n1.in‚ 9408.68359375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.55859375& disk.nvme0n1.out‚79945.57177734375# proc.memory.rssMB‚ 2954.72265625 proc.cpu.threads‚17á(ZA2Á- _wandbruntime‚4109  _runtime‚4109”_ŽÅ¿ _runtime‚ 4109.15446024 _step‚270 & lr-AdamW/pg1‚0.0007411221665479958 & lr-AdamW/pg2‚0.0007411221665479958 trainer/global_step‚3349 $ _timestamp‚1.788533125000911e+09¿‚lÊÃÄ _runtime‚ 4109.15446024 _step‚270 & lr-AdamW/pg1‚0.0007411221665479958 & lr-AdamW/pg2‚0.0007411221665479958 trainer/global_step‚3349 $ _timestamp‚1.788533125000911e+09Ž}±ñž2Ä- _wandbruntime‚4109  _runtime‚4109AA^ùÅÖO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6639353995264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143483980075520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6639120327168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _runtime‚4109.155803642 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63171873 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143484004015616 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90783227 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90756884 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135414250686464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142053353010688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6639120327168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135414232683520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885331250021176e+09 RDDeviceStatsMonitor.on_train_batch_start"76ã/active_bytes.small_pool.peak‚ 565025280 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128960802165872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37667352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90783227 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23983251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 _step‚271 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63171871 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128960819205232 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142053353010688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153955100 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135414232683520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63171871 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6754030577664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150238010653184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6620539508075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37667350 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142053604681728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6754323294720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135581112692867Æ`þ‹á'ÆÛO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63171871 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6754030577664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150238010653184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6620539508075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37667350 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142053604681728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6754323294720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135581112692867 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6639353995264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143483980075520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6639120327168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _runtime‚4109.155803642 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63171873 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143484004015616 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90783227 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90756884 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135414250686464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142053353010688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6639120327168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135414232683520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885331250021176e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128960802165872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37667352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90783227 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23983251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 _step‚271 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63171871 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128960819205232 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142053353010688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153955100 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135414232683520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582¡[…‘Ç‚(:† …³ëÔÃÞÜ™ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.881' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3762 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m${´vÈ‚(:ë …³ëÔ›ÁΛ disk.dm-1.out‚ 130.18359375 disk.sda1.out‚0# disk.nvme0n1.out‚80253.76953125 cpu‚25.288806324149903 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 130.18359375 disk.sda2.out‚0 disk.sda3.out‚ 130.18359375$ memory_percent‚92.05620288848877) proc.memory.availableMB‚ 1301.51171875) proc.memory.percent‚18.03419589996338 network.recv‚ 1.393102e+06( disk./.usagePercent‚78.9946816585225% disk.nvme0c0n1.out‚80253.76953125 disk.sda.in‚0! disk.nvme0n1.in‚ 9408.68359375# proc.memory.rssMB‚ 2954.72265625 proc.cpu.threads‚17 network.sent‚ 7.645694e+06% disk./.usageGB‚1177.0778427124023 disk.dm-0.out‚80253.76953125 disk.dm-1.in‚0 disk.drbd1260.out‚0.5625 disk.sda2.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 9408.68359375 disk.dm-0.in‚ 9408.68359375p¢ç2É- _wandbruntime‚4110  _runtime‚4110võHæ&ÊàM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135453518720512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63190148 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150281692825088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142094854757888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142094603073536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23991009 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23990331 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90783576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63190146 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128998332523296 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚4110.246642837 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6641318034432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6641084353024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143525664045568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚135453536723456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135453536723456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135620601539277 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128998349562656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63190148 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154000093 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6622497997269 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142094854757888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 _step‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61669163 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150281998983168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6756028779520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6622269015981 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61668483 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37678152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚3349 % _timestamp‚1.7885331260923095e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154000093 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61668483 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37678152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚3349 % _timestamp‚1.7885331260923095e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154000093 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135453518720512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63190148 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150281692825088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142094854757888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142094603073536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23991009 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23990331 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90783576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63190146 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128998332523296 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pD™$lÿ ool.peak‚ 6458321920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚4110.246642837 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6641318034432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6641084353024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143525664045568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚135453536723456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135453536723456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135620601539277 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128998349562656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63190148 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154000093 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6622497997269 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273ì;·82Ì- _wandbruntime‚4110  _runtime‚4110XË[\^ÍØ /loss/FBt4/detection_loss‚1.4123118088857547 _runtime‚4110.248229217 0loss/FRt3/assignment_loss‚1.6060320380571727 /loss/FRt1/detection_loss‚2.4335804810890784 1loss/SRqqt2/detection_loss‚2.7446521615982054 1loss/SRqqt3/detection_loss‚0.9794879359779535 .loss/FBt1/detection_loss‚2.520999538898468 /loss/FRt4/assignment_loss‚4.366780714988709 1loss/FBt2/assignment_loss‚0.15731423022755378 .loss/FRt4/detection_loss‚3.247190282344818 0loss/FRt1/assignment_loss‚3.2238892958714414 0loss/FBt1/assignment_loss‚0.3348249658929433 .loss/FRt2/detection_loss‚2.934016949143903 trainer/global_step‚3349 % _timestamp‚1.7885331260947592e+09 2loss/SRqqt1/assignment_loss‚1.6696271176139514 2loss/SRqqt2/assignment_loss‚1.5863415358960629 1loss/SRqqt1/detection_loss‚1.9730016306424751 .loss/FBt3/detection_loss‚0.751283018996841 2loss/SRqqt3/assignment_loss‚0.9931610736099107 1loss/SRqqt4/assignment_loss‚2.259146569917599 0loss/FBt4/assignment_loss‚0.2128254968027225 /loss/FRt3/detection_loss‚1.6048429334737562 /loss/FBt2/detection_loss‚2.2873439037281535 &loss/total_loss‚44.908572644758124 epoch‚2 _step‚273 4loss/FBt3/assignment_loss‚1.4430494551812732e-06 0loss/SRqqt4/detection_loss‚2.423970381418864 0loss/FRt2/assignment_loss‚3.1859471366323273'ù¦cÎÝ /loss/FRt4/assignment_loss‚4.366780714988709 1loss/FBt2/assignment_loss‚0.15731423022755378 .loss/FRt4/detection_loss‚3.247190282344818 0loss/FRt1/assignment_loss‚3.2238892958714414 0loss/FBt1/assignment_loss‚0.3348249658929433 .loss/FRt2/detection_loss‚2.934016949143903 trainer/global_step‚3349 % _timestamp‚1.7885331260947592e+09 2loss/SRqqt1/assignment_loss‚1.6696271176139514 2loss/SRqqt2/assignment_loss‚1.5863415358960629 1loss/SRqqt1/detection_loss‚1.9730016306424751 .loss/FBt3/detection_loss‚0.751283018996841 2loss/SRqqt3/assignment_loss‚0.9931610736099107 1loss/SRqqt4/assignment_loss‚2.259146569917599 0loss/FBt4/assignment_loss‚0.2128254968027225 /loss/FRt3/detection_loss‚1.6048429334737562 /loss/FBt2/detection_loss‚2.2873439037281535 &loss/total_loss‚44.908572644758124 epoch‚2 _step‚273 4loss/FBt3/assignment_loss‚1.4430494551812732e-06 0loss/SRqqt4/detection_loss‚2.423970381418864 0loss/FRt2/assignment_loss‚3.1859471366323273 /loss/FBt4/detection_loss‚1.4123118088857547 _runtime‚4110.248229217 0loss/FRt3/assignment_loss‚1.6060320380571727 /loss/FRt1/detection_loss‚2.4335804810890784 1loss/SRqqt2/detection_loss‚2.7446521615982054 1loss/SRqqt3/detection_loss‚0.9794879359779535 .loss/FBt1/detection_loss‚2.520999538898468‘À[¾‘Ï‚(:† ”³ëÔ¸Œù™ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0358 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ˜pЂ(:å ”³ëÔÛËÏ) disk./.usagePercent‚78.99335579550265# disk.nvme0n1.out‚80354.04296875" disk.nvme0c0n1.in‚ 9418.6171875 disk.dm-0.out‚80354.04296875$ memory_percent‚92.05343723297119# proc.memory.rssMB‚ 2954.72265625 network.sent‚ 7.741713e+06% disk./.usageGB‚1177.0580863952637 disk.sda.in‚0 disk.sda.out‚ 130.6328125 disk.dm-0.in‚ 9418.6171875 disk.dm-1.in‚0 disk.dm-1.out‚ 130.6328125 disk.sda1.out‚0 network.recv‚ 1.39798e+06 disk.sda3.out‚ 130.6328125% disk.nvme0c0n1.out‚80354.04296875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.5625) proc.memory.percent‚18.03419589996338 cpu‚25.288101139575815 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 9418.6171875 disk.sda3.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1301.96484375ìliœÑ‚(:‘ £³ëÔ‰•œ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.686' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.537200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N?﯃Ò‚(:ø £³ëÔž±ó # disk.dm-0.out‚80737.73779296875 network.sent‚ 7.839669e+06 disk.dm-1.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚80737.73779296875 disk.dm-0.in‚ 9533.58203125$ memory_percent‚92.05541610717773# proc.memory.rssMB‚ 2954.80078125 cpu‚25.287502533373992 network.recv‚ 1.404238e+06) disk./.usagePercent‚78.99493203389098 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 9533.58203125( disk.nvme0c0n1.out‚80737.73779296875' proc.memory.availableMB‚ 1301.640625 disk.dm-1.out‚ 130.99609375# disk.nvme0c0n1.in‚ 9533.58203125* proc.memory.percent‚18.034672737121582 proc.cpu.threads‚17% disk./.usageGB‚1177.0815734863281 disk.drbd1260.out‚ 0.56640625 disk.sda.in‚0 disk.sda.out‚ 130.99609375 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 130.99609375Ü‘:ãœÓ‚(:‘ ²³ëԧϞ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.883000000000003 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OOD}…Ô‚(:ú ²³ëԭ別* proc.memory.percent‚18.034672737121582 cpu‚25.286840118812528) disk./.usagePercent‚79.00333343314215 disk.sda.out‚ 131.43359375 disk.sda2.in‚0( disk.nvme0c0n1.out‚80799.93310546875 disk.dm-1.out‚ 131.43359375 disk.sda1.in‚0# proc.memory.rssMB‚ 2954.80078125 proc.cpu.threads‚17 network.sent‚ 7.844646e+06 disk.sda.in‚0& disk.nvme0n1.out‚80799.93310546875 disk.sda3.out‚ 131.43359375# disk.nvme0c0n1.in‚ 9533.58203125 disk.dm-0.in‚ 9533.58203125 disk.dm-1.in‚0 disk.drbd1260.in‚0 network.recv‚ 1.407524e+06% disk./.usageGB‚1177.2067604064941! disk.nvme0n1.in‚ 9533.58203125 disk.sda3.in‚0# disk.dm-0.out‚80798.55810546875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.56640625) proc.memory.availableMB‚ 1302.13671875 disk.sda2.out‚0$ memory_percent‚92.052388191223143†1×2Õ- _wandbruntime‚4167  _runtime‚4167Y~š~ÆÖÀ & lr-AdamW/pg1‚0.0007376730185623614 & lr-AdamW/pg2‚0.0007376730185623614 trainer/global_step‚3399 % _timestamp‚1.7885331832986596e+09 _runtime‚ 4167.45212148 _step‚274=`Q"Ë×Å & lr-AdamW/pg1‚0.0007376730185623614 & lr-AdamW/pg2‚0.0007376730185623614 trainer/global_step‚3399 % _timestamp‚1.7885331832986596e+09 _runtime‚ 4167.45212148 _step‚274’±r}©2Ø- _wandbruntime‚4167  _runtime‚4167ƒÉž,Ü'ÙÖO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145568178577920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137555568604575 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24332481 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152422420314624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38207450 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 _runtime‚4167.453239758 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137378552536064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137378534533120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144116122237952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92094836 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130837320037072 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 _step‚275 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92121179 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24333186 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152422103657472 % _timestamp‚1.7885331832996163e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737569701888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92121179 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130837337076432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38207452 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144115870566912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6737336033792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6718477548583 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚3399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62539931 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6854217796608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144116122237952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64085623 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6853925079552 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92094836 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6737336033792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0où=­á'ÚÛO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 _step‚275 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92121179 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24333186 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152422103657472 % _timestamp‚1.7885331832996163e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737569701888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92121179 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130837337076432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38207452 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144115870566912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6737336033792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6718477548583 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚3399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62539931 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6854217796608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144116122237952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64085623 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6853925079552 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92094836 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6737336033792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145568178577920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137555568604575 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24332481 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152422420314624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38207450 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 _runtime‚4167.453239758 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137378552536064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137378534533120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144116122237952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92094836 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130837320037072 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273“uhÄ2Û- _wandbruntime‚4168  _runtime‚4168vŽ:kÜßM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6855942085632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6739300059648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62558775 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137417838573056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130874850394496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152466110791680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137595303471633 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156225424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720436037777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64103898 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144157120629760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156251795 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6739300059648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚276 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24340521 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62558099 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280:œ¶ê&ÝäM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145609886488064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137595057450985 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130874867433856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137417838573056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚4168.550111361 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6855942085632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6739300059648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62558775 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137417838573056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130874850394496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152466110791680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137595303471633 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156225424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720436037777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64103898 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144157120629760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156251795 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6739300059648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚276 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24340521 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62558099 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B.äçíÖO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63426741 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147652401020416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6816186608011 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64999371 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64999373 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139530024516283 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63427430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24679878 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚Þ( õ 565025280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3449 % _timestamp‚1.7885332422858605e+09 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146178388123136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93459131 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146178639794176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38747552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040žñ>Íá'îÛO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146178388123136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93459131 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146178639794176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38747552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63426741 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147652401020416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6816186608011 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64999371 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64999373 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139530024516283 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63427430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24679878 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3449 % _timestamp‚1.7885332422858605e+09 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63444259 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139569513362693 _runtime‚4227.657364582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158503497 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93485849 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93459480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154649907315712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65017648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158477126 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6837749447680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6837515766272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147694061050368 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24686590 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 % _timestamp‚1.7885332435022914e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38758354 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139382140422656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24685907 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147694084990464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6818145096997 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6837515766272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132751385305056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚3449 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158477126 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93485849 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132751368265696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139382140422656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158503497 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6955846265344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496Èe-„ï ñåM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24686590 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 % _timestamp‚1.7885332435022914e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38758354 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139382140422656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24685907 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147694084990464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6818145096997 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6837515766272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132751385305056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚3449 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158477126 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93485849 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132751368265696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139382140422656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158503497 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6955846265344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65017646 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93459480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38758352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6956132677632 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65017648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63444259 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 F=ÜtuyüDeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139569513362693 _runtime‚4227.657364582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158503497 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93485849 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93459480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154649907315712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65017648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158477126 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6837749447680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6837515766272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147694061050368 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104˜Õ+þj2ò- _wandbruntime‚4227  _runtime‚4227ô—¬]ó× /loss/FRt3/detection_loss‚1.8590421972826856 1loss/FBt2/assignment_loss‚0.24416908120870442 .loss/FRt1/detection_loss‚2.324734916289648 /loss/SRqqt4/detection_loss‚2.61938046344689 epoch‚2 trainer/global_step‚3449 _runtime‚4227.659584126 0loss/FRt1/assignment_loss‚3.4399135126954032 0loss/FRt4/assignment_loss‚3.3915439170339834 1loss/SRqqt2/detection_loss‚2.6840404660805413 /loss/FBt2/detection_loss‚2.1771618057699764 /loss/FBt3/detection_loss‚0.8660961394297316 %loss/total_loss‚44.47733918431677 0loss/SRqqt2/assignment_loss‚1.73178701783004 2loss/SRqqt3/assignment_loss‚0.9014551626751199 2loss/FBt3/assignment_loss‚0.003139523974125991 1loss/FBt4/assignment_loss‚0.09696594333830516 /loss/FRt4/detection_loss‚3.2425866982211238 1loss/FBt1/assignment_loss‚0.18105674297849442 1loss/SRqqt1/detection_loss‚1.9557467830181121 1loss/SRqqt3/detection_loss‚0.9206150898423332 /loss/FRt2/detection_loss‚2.7296224255114794 % _timestamp‚1.7885332435061243e+09 _step‚281 2loss/SRqqt1/assignment_loss‚2.0879732725024223 /loss/FBt4/detection_loss‚1.4743949736961548 /loss/FRt2/assignment_loss‚3.770122744143009 0loss/FRt3/assignment_loss‚1.6062041394046096 .loss/FBt1/detection_loss‚2.814857028032604 2loss/SRqqt4/assignment_loss‚1.3547291399112769ÒEbôÜ 1loss/SRqqt1/detection_loss‚1.9557467830181121 1loss/SRqqt3/detection_loss‚0.9206150898423332 /loss/FRt2/detection_loss‚2.7296224255114794 % _timestamp‚1.7885332435061243e+09 _step‚281 2loss/SRqqt1/assignment_loss‚2.0879732725024223 /loss/FBt4/detection_loss‚1.4743949736961548 /loss/FRt2/assignment_loss‚3.770122744143009 0loss/FRt3/assignment_loss‚1.6062041394046096 .loss/FBt1/detection_loss‚2.814857028032604 2loss/SRqqt4/assignment_loss‚1.3547291399112769 /loss/FRt3/detection_loss‚1.8590421972826856 1loss/FBt2/assignment_loss‚0.24416908120870442 .loss/FRt1/detection_loss‚2.324734916289648 /loss/SRqqt4/detection_loss‚2.61938046344689 epoch‚2 trainer/global_step‚3449 _runtime‚4227.659584126 0loss/FRt1/assignment_loss‚3.4399135126954032 0loss/FRt4/assignment_loss‚3.3915439170339834 1loss/SRqqt2/detection_loss‚2.6840404660805413 /loss/FBt2/detection_loss‚2.1771618057699764 /loss/FBt3/detection_loss‚0.8660961394297316 %loss/total_loss‚44.47733918431677 0loss/SRqqt2/assignment_loss‚1.73178701783004 2loss/SRqqt3/assignment_loss‚0.9014551626751199 2loss/FBt3/assignment_loss‚0.003139523974125991 1loss/FBt4/assignment_loss‚0.09696594333830516 /loss/FRt4/detection_loss‚3.2425866982211238 1loss/FBt1/assignment_loss‚0.18105674297849442™ k/¼‘õ‚(:† ý³ëÔîÙ‚˜ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.226' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4452 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇÉxö‚(:í ý³ëÔ‰¢â™ disk.dm-1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 9691.51171875& disk.nvme0n1.out‚82193.93896484375 proc.cpu.threads‚17 network.recv‚ 1.429975e+06 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.578125 disk.sda2.out‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 9691.51171875( disk.nvme0c0n1.out‚82193.93896484375 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 133.46875 disk.sda3.out‚ 133.46875# memory_percent‚92.0576810836792* proc.memory.percent‚18.034672737121582) disk./.usagePercent‚79.02852969466207 disk.dm-0.in‚ 9691.51171875# disk.dm-0.out‚82193.93896484375 disk.dm-1.out‚ 133.46875) proc.memory.availableMB‚ 1301.26953125# proc.memory.rssMB‚ 2954.80078125 cpu‚25.283694363567516 network.sent‚ 7.974875e+06$ disk./.usageGB‚1177.582202911377E¶|›÷‚(: Œ´ëÔššÀ™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.468000000000004 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R2Þ;ø‚(:ö Œ´ëÔ¥ý’ž disk.drbd1260.out‚0.578125 disk.sda.out‚ 133.91796875' proc.memory.availableMB‚ 1301.515625# proc.memory.rssMB‚ 2954.80078125 cpu‚25.283113402117188 network.sent‚ 8.082149e+06 network.recv‚ 1.435224e+06) disk./.usagePercent‚79.02695320026622% disk./.usageGB‚1177.5587120056152 disk.sda2.out‚0 disk.sda3.out‚ 133.91796875# disk.nvme0c0n1.in‚ 9691.51171875 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 9691.51171875$ memory_percent‚92.05617904663086* proc.memory.percent‚18.034672737121582 proc.cpu.threads‚17 disk.sda2.in‚0! disk.nvme0n1.in‚ 9691.51171875& disk.nvme0n1.out‚82570.96630859375( disk.nvme0c0n1.out‚82570.96630859375# disk.dm-0.out‚82570.96630859375 disk.dm-1.in‚0 disk.dm-1.out‚ 133.91796875 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0òx#Žù‚(:ƒ ›´ëÔàØ–— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.56' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.312 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÿY_{ú‚(:ð ›´ëÔꜣœ$ disk.nvme0n1.out‚82899.115234375 disk.sda3.out‚ 134.38671875$ memory_percent‚92.05625057220459* proc.memory.percent‚18.034672737121582 network.sent‚ 8.090233e+06 network.recv‚ 1.439914e+06) disk./.usagePercent‚79.02845647650727 disk.sda2.out‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 9816.0390625! disk.dm-0.out‚82899.115234375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.58203125 disk.sda.in‚0) proc.memory.availableMB‚ 1301.50390625$ disk./.usageGB‚1177.581111907959& disk.nvme0c0n1.out‚82899.115234375 disk.dm-1.out‚ 134.38671875 disk.sda.out‚ 134.38671875 disk.sda3.in‚0# proc.memory.rssMB‚ 2954.80078125 cpu‚25.282439287033334 disk.dm-0.in‚ 9816.0390625 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 9816.0390625o$Žè‘û‚(:† ª´ëÔžÛÆ• gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7908 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã½ô+yü‚(:î ª´ëÔØÞŠ™# proc.memory.rssMB‚ 2954.80078125* proc.memory.percent‚18.034672737121582% disk./.usageGB‚1177.5810928344727" disk.nvme0c0n1.in‚ 9816.0390625 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 134.8359375 disk.nvme0n1.in‚ 9816.0390625$ disk.nvme0n1.out‚83137.337890625! disk.dm-0.out‚83137.337890625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚92.05477237701416 proc.cpu.threads‚17 network.sent‚ 8.093155e+06 network.recv‚ 1.442903e+06 disk.dm-0.in‚ 9816.0390625 disk.drbd1260.out‚ 0.58203125 disk.sda3.in‚0 cpu‚25.281785860990645) disk./.usagePercent‚79.02845519646961& disk.nvme0c0n1.out‚83137.337890625 disk.dm-1.out‚ 134.8359375 disk.sda.in‚0 disk.sda3.out‚ 134.8359375) proc.memory.availableMB‚ 1301.74609375alÊø2ý- _wandbruntime‚4286  _runtime‚4286Úì[ÇþÁ _runtime‚4286.288440545 _step‚282 & lr-AdamW/pg1‚0.0007305695366992264 & lr-AdamW/pg2‚0.0007305695366992264 trainer/global_step‚3499 % _timestamp‚1.7885333021346612e+09D1·ÌÿÆ % _timestamp‚1.7885333021346612e+09 _runtime‚4286.288440545 _step‚282 & lr-AdamW/pg1‚0.0007305695366992264 & lr-AdamW/pg2‚0.0007305695366992264 trainer/global_step‚3499šÎmêÆ2€- _wandbruntime‚4286  _runtime‚4286qÅ`Û'ÕO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141307156235264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148240905679360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65913123 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148240905679360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148241157350400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149736599522816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7053688218624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64313159 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94797083 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64312442 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6933767447040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65913123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39287650 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141504726448431 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141504480427991 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6914353629599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39287652 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6934001115136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25024792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚4286.290419485 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94770740 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141307138232320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156790263801344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65913121 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94770740 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 trainer/global_step‚3499 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94797083 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6934001115136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 J±Œà'‚ÚO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94770740 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141307138232320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156790263801344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65913121 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94770740 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 trainer/global_step‚3499 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94797083 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6934001115136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141307156235264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148240905679360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65913123 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148240905679360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148241157350400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149736599522816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7053688218624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64313159 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94797083 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64312442 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6933767447040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65913123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39287650 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141504726448431 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141504480427991 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6914353629599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39287652 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6934001115136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25024792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚4286.290419485›jž2ƒ- _wandbruntime‚4287  _runtime‚4287b#ñÿk„àM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148282155742208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160728828 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65931396 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64329929 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6935731472896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141346424269312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39298452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916083137505 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚4287.409719645 _step‚284 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160755199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141346424269312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BÓ-¶{e_segments.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64330630 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94797432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚701 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25031477 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156834251846144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65931396 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160755199 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134627903176256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65931398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885333032550905e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141543969274401 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148282407426560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149778283492864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916312118793 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94823801 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39298454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25032176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65931398 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94823801 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149778259552768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6935965154304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94797432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156833945688064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6935965154304 V”së&…åM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94797432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚701 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25031477 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156834251846144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65931396 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160755199 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134627903176256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65931398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885333032550905e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141543969274401 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148282407426560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149778283492864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916312118793 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94823801 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39298454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25032176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65931398 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94823801 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149778259552768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6935965154304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94797432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156833945688064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6935965154304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148282155742208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160728828 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65931396 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64329929 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6935731472896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141346424269312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39298452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916083137505 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚4287.409719645 _step‚284 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160755199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141346424269312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150303674906624 Vµ”˜rîGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136466890690032 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150303423235584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158974749227520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158974440958976 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7153666873856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7012062689027 trainer/global_step‚3549 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150303674906624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25369029 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7012291670107 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66826873 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7153951202304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143271458084864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66826871 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885333580832648e+09gÔHá'–ÛO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66826873 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7153951202304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143271458084864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66826871 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885333580832648e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7031983153664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39827752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143271458084864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39827750 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143478936339699 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162961908 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150303674906624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136466890690032 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150303423235584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158974749227520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158974440958976 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7153666873856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7012062689027 trainer/global_step‚3549 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150303674906624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25369029 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7012291670107 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608Ÿî‹’2—- _wandbruntime‚4343  _runtime‚4343–¨â¥æ&˜àM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96161753 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66845146 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65214647 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151862458055168 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159018115800064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39838552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7155952545792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143518671206757 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136504421047456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7155657744896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136504404008096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66845148 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150344924982784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163006901 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96135384 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66845146 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159018434541056 _runtime‚4343.303320607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143310744121856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25376745 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7034180860928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151862481995264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163006901 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162980530 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1433107441218567…ôfö!™åM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66845146 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65214647 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151862458055168 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159018115800064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39838552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7155952545792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143518671206757 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136504421047456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7155657744896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136504404008096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66845148 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150344924982784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163006901 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96135384 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66845146 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159018434541056 _runtime‚4343.303320607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143310744121856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25376745 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7034180860928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151862481995264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163006901 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162980530 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143310744121856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96161753 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 lìÇì2š- _wandbruntime‚4343  _runtime‚4343ëÈ/@Z›Ô 3loss/FBt3/assignment_loss‚0.0032703466766244153 1loss/SRqqt1/detection_loss‚1.4406540263444185 1loss/SRqqt4/detection_loss‚2.5697218564649424 /loss/FBt4/detection_loss‚1.3165582355111838 %loss/total_loss‚44.71000551988619 epoch‚2 0loss/FRt4/assignment_loss‚3.5339155544837317 -loss/FBt2/detection_loss‚2.10103602334857 /loss/FRt3/assignment_loss‚1.411031817460989 1loss/FBt1/assignment_loss‚0.27879697461516784 /loss/FRt2/assignment_loss‚3.852303531434801 1loss/SRqqt2/assignment_loss‚1.962759254943757 2loss/SRqqt4/assignment_loss‚1.9775962630907695 .loss/FRt1/detection_loss‚2.327423860346535 /loss/SRqqt2/detection_loss‚2.83252861953917 /loss/FBt3/detection_loss‚0.8957544383193765 _runtime‚4343.305318483 0loss/FBt2/assignment_loss‚0.2372177042519752 2loss/SRqqt3/assignment_loss‚1.1305063428477413 /loss/FBt1/detection_loss‚2.6659891988549913 trainer/global_step‚3549 % _timestamp‚1.7885333591519358e+09 _step‚289 0loss/FRt1/assignment_loss‚3.6418428352900913 2loss/SRqqt1/assignment_loss‚1.3301362074645502 0loss/SRqqt3/detection_loss‚0.988704178842806 0loss/FBt4/assignment_loss‚0.1732765018450531 /loss/FRt2/detection_loss‚3.1922239330079822 /loss/FRt3/detection_loss‚1.5179332451193364 /loss/FRt4/detection_loss‚3.3288245697816214N­6_œÙ 0loss/FRt4/assignment_loss‚3.5339155544837317 -loss/FBt2/detection_loss‚2.10103602334857 /loss/FRt3/assignment_loss‚1.411031817460989 1loss/FBt1/assignment_loss‚0.27879697461516784 /loss/FRt2/assignment_loss‚3.852303531434801 1loss/SRqqt2/assignment_loss‚1.962759254943757 2loss/SRqqt4/assignment_loss‚1.9775962630907695 .loss/FRt1/detection_loss‚2.327423860346535 /loss/SRqqt2/detection_loss‚2.83252861953917 /loss/FBt3/detection_loss‚0.8957544383193765 _runtime‚4343.305318483 0loss/FBt2/assignment_loss‚0.2372177042519752 2loss/SRqqt3/assignment_loss‚1.1305063428477413 /loss/FBt1/detection_loss‚2.6659891988549913 trainer/global_step‚3549 % _timestamp‚1.7885333591519358e+09 _step‚289 0loss/FRt1/assignment_loss‚3.6418428352900913 2loss/SRqqt1/assignment_loss‚1.3301362074645502 0loss/SRqqt3/detection_loss‚0.988704178842806 0loss/FBt4/assignment_loss‚0.1732765018450531 /loss/FRt2/detection_loss‚3.1922239330079822 /loss/FRt3/detection_loss‚1.5179332451193364 /loss/FRt4/detection_loss‚3.3288245697816214 3loss/FBt3/assignment_loss‚0.0032703466766244153 1loss/SRqqt1/detection_loss‚1.4406540263444185 1loss/SRqqt4/detection_loss‚2.5697218564649424 /loss/FBt4/detection_loss‚1.3165582355111838 %loss/total_loss‚44.71000551988619 epoch‚2¡E|b ‘‚(:† õ´ëÔ²²˜ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5612 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ûxž‚(:í õ´ëÔ˜÷Žš disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 137.31640625 cpu‚25.27877821585709% disk./.usageGB‚1175.4212646484375 disk.sda.in‚0% disk.nvme0n1.out‚84637.8388671875 disk.sda3.out‚ 137.31640625 disk.dm-0.in‚ 9968.3046875" disk.dm-0.out‚84637.8388671875 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 8.22485e+06 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 9968.3046875 disk.drbd1260.out‚0.59375$ memory_percent‚92.05694198608398# proc.memory.rssMB‚ 2954.80078125 network.recv‚ 1.466684e+06 disk.nvme0n1.in‚ 9968.3046875' disk.nvme0c0n1.out‚84637.8388671875 disk.sda1.out‚0 disk.drbd1260.in‚0' proc.memory.availableMB‚ 1301.390625* proc.memory.percent‚18.034672737121582) disk./.usagePercent‚78.88350731468823 disk.sda.out‚ 137.31640625ÇZ˜~‘Ÿ‚(:† „µëԆײ— gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3736 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íFt ‚(:é „µëÔü÷áš disk.dm-0.in‚ 9968.3046875 disk.drbd1260.out‚0.59375 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0* proc.memory.percent‚18.034672737121582 network.recv‚ 1.472167e+06' disk.nvme0c0n1.out‚84849.0810546875 disk.dm-1.out‚ 137.765625 disk.sda.out‚ 137.765625 disk.nvme0n1.in‚ 9968.3046875 disk.sda3.out‚ 137.765625$ memory_percent‚92.05546379089355 network.sent‚ 8.332343e+06( disk./.usagePercent‚78.8803586780246" disk.dm-0.out‚84849.0810546875( proc.memory.availableMB‚ 1301.6328125# proc.memory.rssMB‚ 2954.80078125 proc.cpu.threads‚17% disk./.usageGB‚1175.3743476867676 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚84849.0810546875 cpu‚25.278160794001536" disk.nvme0c0n1.in‚ 9968.3046875ó–-™œ¡‚(:‘ “µëÔô£¼— gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚130.396' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.079199999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mw(^r¢‚(:ç “µëÔÓžäš& disk.nvme0c0n1.out‚85012.310546875 network.recv‚ 1.476879e+06 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 138.09375# disk.nvme0c0n1.in‚ 10056.4609375# memory_percent‚92.0569896697998# proc.memory.rssMB‚ 2954.80078125 cpu‚25.277612497917918 disk.sda1.in‚0 disk.sda.out‚ 138.09375 disk.dm-0.in‚ 10056.4609375 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 8.338557e+06 disk.sda1.out‚0 disk.drbd1260.in‚0! disk.dm-0.out‚85012.310546875 disk.dm-1.out‚ 138.09375( proc.memory.availableMB‚ 1301.3828125* proc.memory.percent‚18.034672737121582' disk./.usagePercent‚78.880357910002% disk./.usageGB‚1175.3743362426758 disk.drbd1260.out‚ 0.59765625 disk.sda.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 10056.4609375$ disk.nvme0n1.out‚85012.310546875ùG‘£‚(:† ¢µëÔêâ– gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.479' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8958 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)ìl}¤‚(:ò ¢µëÔî²ä— disk.drbd1260.in‚0 disk.sda.out‚ 138.59765625 disk.sda2.in‚0! disk.nvme0n1.in‚ 10162.9609375* proc.memory.percent‚18.034672737121582 proc.cpu.threads‚17% disk./.usageGB‚1175.2967529296875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 138.59765625$ memory_percent‚92.05546379089355( proc.memory.availableMB‚ 1301.6328125 cpu‚25.27699019062502 network.recv‚ 1.480046e+06& disk.nvme0c0n1.out‚85104.080078125 disk.sda1.out‚0 disk.drbd1260.out‚ 0.59765625 disk.sda.in‚0# proc.memory.rssMB‚ 2954.80078125 network.sent‚ 8.343496e+06) disk./.usagePercent‚78.87515122877062# disk.nvme0c0n1.in‚ 10162.9609375! disk.dm-0.out‚85104.080078125 disk.dm-1.in‚0 disk.dm-1.out‚ 138.59765625$ disk.nvme0n1.out‚85104.080078125 disk.dm-0.in‚ 10162.9609375¡$2¥- _wandbruntime‚4398  _runtime‚4398l‘fƦÀ & lr-AdamW/pg1‚0.0007231974315938314 & lr-AdamW/pg2‚0.0007231974315938314 trainer/global_step‚3599 % _timestamp‚1.7885334140955002e+09 _runtime‚ 4398.24914125 _step‚290ÕñÝy˧Š& lr-AdamW/pg1‚0.0007231974315938314 & lr-AdamW/pg2‚0.0007231974315938314 trainer/global_step‚3599 % _timestamp‚1.7885334140955002e+09 _runtime‚ 4398.24914125 _step‚290¢#dhã2¨- _wandbruntime‚4398  _runtime‚4398¾Û'©ÕO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7253509163008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165213610 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138343408561232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165187265 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40367850 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25717744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67740623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145235759934464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145235741931520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97472987 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7110000729535 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153904972587520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138343391521872 _runtime‚4398.250376789 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152365940791808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153904996527616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 _step‚291 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97446644 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7130432528384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67740621 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚3599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67740621 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25718467 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145453638271847 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7110229710615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7130198860288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152366192462848 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145235759934464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145235741931520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0éª9tà'ªÚO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40367852 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161158481750528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7253799782912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 $ _timestamp‚1.788533414096701e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97472987 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152366192462848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7130198860288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161158796310528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152365940791808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66086319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153904972587520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138343391521872 _runtime‚4398.250376789 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152365940791808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153904996527616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 _step‚291 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97446644 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7130432528384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67740621 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚3599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67740621 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25718467 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145453638271847 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7110229710615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7130198860288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152366192462848 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145235759934464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145235741931520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7253509163008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165213610 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138343408561232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165187265 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40367850 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25717744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67740623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145235759934464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145235741931520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97472987 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7110000729535 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97499705 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7132162886144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145275045971456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145275045971456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66103020 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25725007 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161202469525504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138380938918656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165232232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161202159173120 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25724368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255502615552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67758896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152407442539008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145275027968512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7885334150472858e+09 _step‚292 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145275027968512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67758898 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40378652 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7255789027840 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165258603 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66103661 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153946680497664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165258603 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67758898 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145492881097817 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7132396567552Zž¿ë&­åM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97499705 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97473336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40378654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7132396567552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97499705 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7132162886144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145275045971456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145275045971456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66103020 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25725007 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161202469525504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138380938918656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165232232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161202159173120 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25724368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255502615552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67758896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152407442539008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145275027968512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7885334150472858e+09 _step‚292 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145275027968512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67758898 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40378652 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7255789027840 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165258603 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66103661 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153946680497664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165258603 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67758898 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145492881097817 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7132396567552¤:d2®- _wandbruntime‚4399  _runtime‚4399úë0[¯Õ 2loss/SRqqt3/assignment_loss‚1.0441123266897068 0loss/FBt2/assignment_loss‚0.6853188071399927 /loss/FRt3/detection_loss‚1.4571320209887964 0loss/SRqqt2/detection_loss‚2.270625736564398 0loss/SRqqt4/detection_loss‚2.240909284160983 /loss/FBt2/detection_loss‚2.6275613367557527 /loss/FRt1/assignment_loss‚3.346040204951638 2loss/SRqqt2/assignment_loss‚1.8835166955832392 1loss/FBt4/assignment_loss‚0.30841404334992883 0loss/FRt3/assignment_loss‚1.6136613209036332 %loss/total_loss‚45.34473248463864 1loss/SRqqt4/assignment_loss‚1.993451167018183 4loss/FBt3/assignment_loss‚5.5433820294581436e-05 /loss/FRt2/detection_loss‚2.9397246329872697 /loss/FBt1/detection_loss‚2.9655815958976746 epoch‚2 trainer/global_step‚3599 % _timestamp‚1.7885334150500803e+09 _runtime‚4399.203439776 1loss/SRqqt1/assignment_loss‚1.409670812357217 1loss/SRqqt1/detection_loss‚2.0957017600536347 /loss/FBt3/detection_loss‚0.7757911045518187 _step‚293 /loss/FRt4/assignment_loss‚3.965677399635315 1loss/FBt1/assignment_loss‚0.26867044798564077 .loss/FRt1/detection_loss‚2.484401040955596 -loss/FRt4/detection_loss‚3.15072208404541 0loss/SRqqt3/detection_loss‚0.945691449760363 0loss/FRt2/assignment_loss‚3.5913079977035522 /loss/FBt4/detection_loss‚1.2809937807786114›[ïh`°Ú 1loss/SRqqt4/assignment_loss‚1.993451167018183 4loss/FBt3/assignment_loss‚5.5433820294581436e-05 /loss/FRt2/detection_loss‚2.9397246329872697 /loss/FBt1/detection_loss‚2.9655815958976746 epoch‚2 trainer/global_step‚3599 % _timestamp‚1.7885334150500803e+09 _runtime‚4399.203439776 1loss/SRqqt1/assignment_loss‚1.409670812357217 1loss/SRqqt1/detection_loss‚2.0957017600536347 /loss/FBt3/detection_loss‚0.7757911045518187 _step‚293 /loss/FRt4/assignment_loss‚3.965677399635315 1loss/FBt1/assignment_loss‚0.26867044798564077 .loss/FRt1/detection_loss‚2.484401040955596 -loss/FRt4/detection_loss‚3.15072208404541 0loss/SRqqt3/detection_loss‚0.945691449760363 0loss/FRt2/assignment_loss‚3.5913079977035522 /loss/FBt4/detection_loss‚1.2809937807786114 2loss/SRqqt3/assignment_loss‚1.0441123266897068 0loss/FBt2/assignment_loss‚0.6853188071399927 /loss/FRt3/detection_loss‚1.4571320209887964 0loss/SRqqt2/detection_loss‚2.270625736564398 0loss/SRqqt4/detection_loss‚2.240909284160983 /loss/FBt2/detection_loss‚2.6275613367557527 /loss/FRt1/assignment_loss‚3.346040204951638 2loss/SRqqt2/assignment_loss‚1.8835166955832392 1loss/FBt4/assignment_loss‚0.30841404334992883 0loss/FRt3/assignment_loss‚1.6136613209036332 %loss/total_loss‚45.34473248463864¥qô´Ö±‚(:‚ ±µëÔûøÒ˜ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.55' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.11 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍÃp‡x²‚(:í ±µëÔ‹«â™ disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 network.recv‚ 1.485243e+06& disk.nvme0c0n1.out‚85189.189453125 disk.dm-1.out‚ 139.1171875 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 1301.5859375 proc.cpu.threads‚17) disk./.usagePercent‚78.86966575533447 disk.dm-0.in‚ 10162.9609375! disk.dm-0.out‚85189.189453125 disk.sda1.in‚0 disk.sda.out‚ 139.1171875! disk.nvme0n1.in‚ 10162.9609375$ disk.nvme0n1.out‚85189.189453125 cpu‚25.27636975605147 disk.sda1.out‚0 disk.drbd1260.out‚ 0.6015625 disk.sda3.out‚ 139.1171875$ memory_percent‚92.05574989318848# proc.memory.rssMB‚ 2954.80078125* proc.memory.percent‚18.034672737121582 network.sent‚ 8.349886e+06$ disk./.usageGB‚1175.215015411377# disk.nvme0c0n1.in‚ 10162.9609375@Q³‚(:… ÀµëÔµÌÚ• gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚139.035' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.807 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈrK¨v´‚(:ë ÀµëÔÄÎÛ™' proc.memory.availableMB‚ 1301.578125% disk./.usageGB‚1179.2946128845215 disk.sda1.in‚0* proc.memory.percent‚18.034744262695312 proc.cpu.threads‚17& disk.nvme0c0n1.out‚85615.458984375! disk.dm-0.out‚85572.708984375 network.recv‚ 1.490345e+06) disk./.usagePercent‚79.14345096476747 disk.sda2.in‚0 disk.sda3.out‚ 139.50390625 disk.dm-1.out‚ 139.50390625 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 10162.9609375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.6015625 disk.sda.in‚0 disk.sda.out‚ 139.50390625 proc.memory.rssMB‚ 2954.8125 cpu‚25.27572148310351 disk.dm-0.in‚ 10162.9609375 network.sent‚ 8.457244e+06! disk.nvme0n1.in‚ 10162.9609375$ disk.nvme0n1.out‚85615.458984375 disk.sda3.in‚0 disk.dm-1.in‚0# memory_percent‚92.0557975769043~-¿Q‘µ‚(:† ϵëÔÖª†— gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.648' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5296 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø2w¶‚(:ì ϵëÔ¨ÖÏš disk.drbd1260.out‚ 0.68359375) proc.memory.availableMB‚ 1216.88671875 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 18234.7109375 network.sent‚ 8.463289e+06( disk./.usagePercent‚79.5466559189925% disk./.usageGB‚1185.3026580810547! disk.nvme0n1.in‚ 18234.7109375 disk.sda1.out‚0" proc.memory.rssMB‚ 3039.0390625( proc.memory.percent‚18.5488224029541 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚95703.76904296875( disk.nvme0c0n1.out‚95703.76904296875 disk.sda1.in‚0# memory_percent‚92.5727128982544 cpu‚25.284840199423506 network.recv‚ 1.494932e+06 disk.sda.out‚ 139.90625 disk.sda3.in‚0 disk.sda3.out‚ 139.90625 disk.dm-0.in‚ 18234.7109375# disk.dm-0.out‚95703.76904296875 disk.dm-1.in‚0 disk.dm-1.out‚ 139.90625 disk.sda.in‚0 disk.drbd1260.in‚0®š/Ë·‚(:… ÞµëÔÀ¶Õ™ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜xÁry¸‚(:î ÞµëÔîïÚž disk.sda.out‚ 140.3203125 disk.sda2.out‚0( disk.nvme0c0n1.out‚95728.95654296875# disk.dm-0.out‚95728.95654296875 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 18801.96875 disk.sda3.out‚ 140.3203125 disk.dm-0.in‚ 18801.96875 disk.sda1.in‚0$ memory_percent‚92.56923198699951 network.recv‚ 1.498471e+06) disk./.usagePercent‚78.87743635202126 disk.sda2.in‚0& disk.nvme0n1.out‚95728.95654296875 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 18801.96875 disk.dm-1.out‚ 140.3203125( proc.memory.percent‚18.5488224029541% disk./.usageGB‚1175.3308029174805 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1217.45703125" proc.memory.rssMB‚ 3039.0390625 cpu‚25.296310931829805 proc.cpu.threads‚17 network.sent‚ 8.470024e+06 disk.drbd1260.out‚ 0.68359375åFÔÈ‘¹‚(:† íµëÔÃîÊ• gpu.0.gpu‚16.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5542 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i)Aï{º‚(:ð íµëÔ昼œ) disk./.usagePercent‚78.88801586337374" disk.dm-0.out‚96233.4306640625 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚18814.37109375 disk.dm-1.in‚0 disk.dm-1.out‚ 140.83984375 disk.sda1.out‚0 disk.sda3.out‚ 140.83984375" proc.memory.rssMB‚ 3039.0390625( proc.memory.percent‚18.5488224029541 proc.cpu.threads‚17 network.recv‚ 1.5034e+06% disk./.usageGB‚1175.4884452819824 disk.drbd1260.out‚0.6875 disk.sda.out‚ 140.83984375" disk.nvme0n1.in‚18814.37109375$ memory_percent‚92.57075786590576) proc.memory.availableMB‚ 1217.20703125 cpu‚25.303838222645087 network.sent‚ 8.47651e+06% disk.nvme0n1.out‚96233.4306640625$ disk.nvme0c0n1.in‚18814.37109375' disk.nvme0c0n1.out‚96233.4306640625;˜%Ç»‚(:… üµëÔ×–³• gpu.0.gpu‚8.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0võÛý}¼‚(:ò üµëÔÇí—) disk./.usagePercent‚78.88802021550183 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.6875" disk.nvme0n1.in‚18814.37109375) proc.memory.availableMB‚ 1216.94921875( proc.memory.percent‚18.5488224029541 network.sent‚ 8.479474e+06 network.recv‚ 1.506777e+06$ disk./.usageGB‚1175.488510131836 disk.dm-1.in‚0 disk.dm-1.out‚ 141.32421875 disk.sda.out‚ 141.32421875 disk.sda2.in‚0% disk.nvme0n1.out‚96510.5263671875$ disk.nvme0c0n1.in‚18814.37109375' disk.nvme0c0n1.out‚96510.5263671875 disk.dm-0.in‚18814.37109375" disk.dm-0.out‚96510.5263671875 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 141.32421875 disk.sda.in‚0$ memory_percent‚92.57233142852783" proc.memory.rssMB‚ 3039.0390625 cpu‚25.313224134786687 proc.cpu.threads‚17j‚/›½‚(: ‹¶ëÔ˜À¼˜ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.684800000000003 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚­ƒ¾‚(:ø ‹¶ëÔéØÔž$ disk.nvme0c0n1.in‚18814.37109375 disk.sda2.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 3039.0390625( proc.memory.percent‚18.5488224029541 cpu‚25.319880805023402 proc.cpu.threads‚17% disk./.usageGB‚1175.4649658203125 disk.dm-0.in‚18814.37109375# disk.dm-0.out‚96848.97900390625 disk.dm-1.out‚ 141.71484375 disk.sda1.in‚0 disk.drbd1260.out‚0.703125 disk.sda.out‚ 141.71484375" disk.nvme0n1.in‚18814.37109375 network.sent‚ 8.485589e+06 network.recv‚ 1.511434e+06) disk./.usagePercent‚78.88644013700049( disk.nvme0c0n1.out‚96848.97900390625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚96848.97900390625 disk.sda3.out‚ 141.71484375$ memory_percent‚92.57495403289795) proc.memory.availableMB‚ 1216.51953125¹‰Š{2¿- _wandbruntime‚4511  _runtime‚4511Ù ¶™² À½T LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32005 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151344128 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚µÿ‘163892833606144 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚147834719029451 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20710 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚98636618 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚155017457387008 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚147834907889279 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚41242726 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059453 % _timestamp‚1.7885335278100839e+09 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚69006041 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900224 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚69006039 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚155017651201536 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193814528 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 167663369 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚67451917 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚294 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820468 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167642657 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚147808969005056 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚67452310 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚393 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859828 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7332959990784 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚69006039 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025280 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚26209193 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 175284224 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025280 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚140646681401072 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288773 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7208664193536 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 7208488381952 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7332808646656 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 167663369 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41242724 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7208664193536 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20710 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1îUíH*ÁÂT OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32248 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚156559849675264 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚4511.965358583 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193814528 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20712 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚163892658321920 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32005 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151344128 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚163892833606144 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚147834719029451 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20710 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚98636618 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚155017457387008 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚147834907889279 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚41242726 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059453 % _timestamp‚1.7885335278100839e+09 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚69006041 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900224 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚69006039 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚155017651201536 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193814528 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 167663369 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚67451917 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚294 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820468 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167642657 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚147808969005056 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚67452310 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚393 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859828 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7332959990784 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚69006039 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025280 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚26209193 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 175284224 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025280 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚140646681401072 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288773 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7208664193536 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 7208488381952 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7332808646656 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 167663369 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41242724 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7208664193536 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20710 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193814528 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7209138053632 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚155048190735872 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚147839052682240 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚391 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859828 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288773 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚156590211428864 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820468 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7208962242048 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67469776 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚147860939746723 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059453 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚140672241050704._3…ÄÌR ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7209138053632 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚155048190735872 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚147839052682240 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚391 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859828 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288773 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚156590211428864 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820468 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7208962242048 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67469776 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚147860939746723 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059453 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚140672241050704 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344128 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 7188698696019 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20710 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚155048190735872 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900224 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚393 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚147860750886895 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚41254406 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26215761 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚98666236 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 167687659 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025280 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7209138053632 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚147839034679296 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚98645526 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 167666947 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚67470169 _runtime‚4513.335653712 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all÷Æk¬Í.peak‚ 347418624 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚41254408 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 _step‚295 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175811584 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚163923669219840 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284224 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193814528 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104§G”6þ›Å‚(: š¶ëÔ®šÖš gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.077' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.415399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š`î.wÆ‚(:ì š¶ëÔ«ÒÜœ disk.dm-0.in‚ 18957.0859375# disk.dm-0.out‚97200.97509765625 disk.dm-1.in‚0 disk.dm-1.out‚ 142.265625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.703125 disk.sda2.in‚0% disk./.usageGB‚1175.5511169433594 disk.sda.in‚0 disk.sda.out‚ 142.265625 disk.sda2.out‚0& disk.nvme0n1.out‚97200.97509765625 disk.sda3.in‚0 disk.sda3.out‚ 142.265625( disk.nvme0c0n1.out‚97200.97509765625 network.sent‚ 8.490251e+06) disk./.usagePercent‚78.89222181115377 disk.sda1.in‚0( proc.memory.availableMB‚ 1218.5390625 cpu‚25.32923104737946 network.recv‚ 1.513983e+06! disk.nvme0n1.in‚ 18957.0859375# disk.nvme0c0n1.in‚ 18957.0859375# memory_percent‚92.5626277923584" proc.memory.rssMB‚ 3039.0390625( proc.memory.percent‚18.5488224029541 proc.cpu.threads‚17Ô¹b©œÇ‚(:‘ ©¶ëÔÒ›–™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚140.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.084799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö÷#€È‚(:õ ©¶ëÔ Ãè¡ cpu‚25.340793921447993% disk./.usageGB‚1175.5410270690918 disk.sda.in‚0( disk.nvme0c0n1.out‚97201.83544921875 disk.dm-1.out‚ 142.70703125 disk.sda1.in‚0( disk./.usagePercent‚78.8915446712257 disk.sda.out‚ 142.70703125 disk.sda3.out‚ 142.70703125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0( proc.memory.percent‚18.5488224029541 network.sent‚ 8.600041e+06 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 18957.0859375 disk.dm-0.in‚ 18957.0859375 disk.drbd1260.out‚ 0.70703125( proc.memory.availableMB‚ 1218.9765625" proc.memory.rssMB‚ 3039.0390625 proc.cpu.threads‚17 network.recv‚ 1.520681e+06 disk.sda2.out‚0! disk.nvme0n1.in‚ 18957.0859375& disk.nvme0n1.out‚97201.83544921875 disk.sda3.in‚0# disk.dm-0.out‚97201.83544921875$ memory_percent‚92.55995750427246÷W2É- _wandbruntime‚4530  _runtime‚4530y©Ù¢Zʧ¥ particle/accuracy_3_of_3‚1 7validation_average_jet_accuracy‚0.42296531796455383 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5079861283302307 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6666666269302368 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8138357996940613 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007061119191348553 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4354710280895233 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17003534734249115 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1197916641831398 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5304923057556152 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9795044660568237 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.492407888174057 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.973958432674408 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 _runtime‚4530.770651588 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6359511613845825 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38487643003463745 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5880626440048218 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.36913979053497314 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3821541368961334 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21007448434829712 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4611279368400574 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.38446399569511414 *jet/accuracy_2_of_5‚0.4715181291103363 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2545662224292755 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4131944477558136 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9891442656517029 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 particle/accuracy_3_of_7‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01816578581929207 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47333332896232605 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09900228679180145 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.47256943583488464 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44466549158096313 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.25058799982070923 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47165513038635254 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016843033954501152 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2732687294483185 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_6‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01872568018734455 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5553240180015564 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021802494302392006 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34153977036476135 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4956139922142029 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2300652414560318 particle/accuracy_1_of_6‚1 particle/accuracy_5_of_7‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1574983298778534 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3929910957813263 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00925220176577568 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42996862530708313 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4128113389015198 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.290640652179718 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.010062893852591515 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 *jet/accuracy_1_of_1‚0.4646221697330475 particle/accuracy_3_of_4‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04183848947286606 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9748655557632446 *jet/accuracy_1_of_4‚0.8998486399650574 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22047917544841766 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.27096235752105713 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3338955342769623 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4252488315105438 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3594643771648407 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5242854952812195 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.42626461386680603 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 jet/accuracy_1_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.27727001905441284 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4399225115776062 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35092592239379883 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473652839661 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4094456136226654 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45062679052352905 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9495954513549805 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23652885854244232 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2717472314834595 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt2FBt/SRqqt_purity‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5099207162857056 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.40120694041252136 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4162936806678772 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2797619104385376 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4088922441005707 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5376302599906921 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9950212240219116 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.018880076706409454 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.28320881724357605 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38556551933288574 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9840697050094604 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.36647650599479675 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04945584759116173 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19820554554462433 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5339694619178772 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.973611056804657 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8502809405326843 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.44878044724464417 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.040592044591903687 ,particle/sensitivity‚0.31357845664024353 epoch‚2 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5148489475250244 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5137388110160828 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973068237304688 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 +jet/accuracy_3_of_4‚0.06557126343250275 particle/accuracy_4_of_5‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5455648899078369 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.470631867647171 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4527410864830017 ,jet/accuracy_4_of_4‚0.005029045976698399 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.46800851821899414 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5467749238014221 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.44828805327415466 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 +jet/accuracy_2_of_3‚0.24178561568260193 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4439953863620758 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.375 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09434203058481216 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166567325592 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981160163879395 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14358574151992798 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0798070877790451 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19166401028633118 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3057471215724945 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 trainer/global_step‚3626 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.026672102510929108 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9904367923736572 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19177688658237457 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2578168213367462 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2708333432674408 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2545199990272522 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2471264749765396 jet/accuracy_4_of_5‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606059432029724 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2936026453971863 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_2‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.31257307529449463 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3471677899360657 _step‚296 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.25795233249664307 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.314414381980896 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.22839508950710297 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4508272707462311 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2637229561805725 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.24271857738494873 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3509804606437683 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19197367131710052 *jet/accuracy_1_of_3‚0.8572184443473816 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36000001430511475 jet/accuracy_2_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 particle/accuracy_1_of_4‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.35407450795173645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.27142226696014404 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172577261924744 (particle/f_score‚0.39531001448631287 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.44015243649482727 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26934024691581726 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4340665936470032 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 6 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.225917249917984 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3819533884525299 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24211180210113525 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978061318397522 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5137161016464233 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3888303339481354 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9890904426574707 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2826901376247406 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5172618627548218 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_1_of_1‚1 particle/accuracy_4_of_6‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/*FRt3SRqqt0FB“@K) 8t/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17791956663131714 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5094322562217712 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.036098312586545944 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 jet/accuracy_6_of_7‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5358009934425354 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0017636684933677316 +jet/accuracy_3_of_6‚0.22435897588729858 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 particle/accuracy_5_of_6‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3663793206214905 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10052461922168732 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9815800189971924 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05747126415371895 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0012210012646391988 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0021919880528002977 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9894683957099915 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_6‚1 particle/accuracy_2_of_7‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4242244362831116 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2209424078464508 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3057681918144226 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23986487090587616 *jet/accuracy_1_of_2‚0.7337465882301331 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3915874660015106 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9798322916030884 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9865179061889648 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04434523731470108 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27609318494796753 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17791956663131714 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24747473001480103 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10768435150384903 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2709551155567169 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4030773937702179 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.46543610095977783 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15175525844097137 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4456075131893158 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10263795405626297 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5159482359886169 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 )loss/val_total_loss‚43.82512802954303 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991101622581482 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2370029091835022 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.987169086933136 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906829595565796 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9788074493408203 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4224702715873718 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.2111111432313919 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1819489747285843 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.022175883874297142 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5448717474937439 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.30124524235725403 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16948235034942627 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2430555820465088 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 jet/accuracy_4_of_6‚0 particle/accuracy_6_of_6‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3138718008995056 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2776981294155121 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9921806454658508 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_6‚0.9807692170143127 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.32683029770851135 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.38294368982315063 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26552626490592957 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5525793433189392 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25462964177131653 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.48842594027519226 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4922565221786499 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977529644966125 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22175665199756622 jet/accuracy_5_of_7‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 $ _timestamp‚1.788533546615181e+09 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347031593323 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 *jet/accuracy_1_of_5‚0.9308307766914368 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 jet/accuracy_3_of_7‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6997992396354675 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.45268958806991577 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0035714285913854837 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2527502179145813 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21436777710914612 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.138888880610466 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24033373594284058 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9893429279327393 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4222588539123535 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01205774862319231 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.36357852816581726 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_6‚1 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00515872985124588 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9906359314918518 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.980571448802948 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45062679052352905 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043½"JnæGˬ¥ 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3819533884525299 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24211180210113525 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978061318397522 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5137161016464233 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3888303339481354 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9890904426574707 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2826901376247406 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5172618627548218 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_1_of_1‚1 particle/accuracy_4_of_6‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17791956663131714 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5094322562217712 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.036098312586545944 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 jet/accuracy_6_of_7‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5358009934425354 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0017636684933677316 +jet/accuracy_3_of_6‚0.22435897588729858 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 particle/accuracy_5_of_6‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3663793206214905 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10052461922168732 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9815800189971924 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05747126415371895 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0012210012646391988 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0021919880528002977 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9894683957099915 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_6‚1 particle/accuracy_2_of_7‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4242244362831116 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2209424078464508 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3057681918144226 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23986487090587616 *jet/accuracy_1_of_2‚0.7337465882301331 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3915874660015106 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9798322916030884 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9865179061889648 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04434523731470108 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27609318494796753 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17791956663131714 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24747473001480103 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10768435150384903 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2709551155567169 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4030773937702179 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.46543610095977783 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15175525844097137 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4456075131893158 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10263795405626297 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5159482359886169 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 )loss/val_total_loss‚43.82512802954303 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991101622581482 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2370029091835022 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.987169086933136 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906829595565796 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9788074493408203 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4224702715873718 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.2111111432313919 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1819489747285843 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.022175883874297142 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5448717474937439 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.30124524235725403 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16948235034942627 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2430555820465088 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 jet/accuracy_4_of_6‚0 particle/accuracy_6_of_6‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3138718008995056 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2776981294155121 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9921806454658508 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_6‚0.9807692170143127 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.32683029770851135 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.38294368982315063 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26552626490592957 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5525793433189392 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25462964177131653 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.48842594027519226 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4922565221786499 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977529644966125 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22175665199756622 jet/accuracy_5_of_7‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 $ _timestamp‚1.788533546615181e+09 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347031593323 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 *jet/accuracy_1_of_5‚0.9308307766914368 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 jet/accuracy_3_of_7‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6997992396354675 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.45268958806991577 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0035714285913854837 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2527502179145813 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21436777710914612 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.138888880610466 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24033373594284058 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9893429279327393 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4222588539123535 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01205774862319231 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.36357852816581726 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_6‚1 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00515872985124588 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9906359314918518 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.980571448802948 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45062679052352905 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 particle/accuracy_3_of_3‚1 7validation_average_jet_accuracy‚0.42296531796455383 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5079861283302307 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6666666269302368 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8138357996940613 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007061119191348553 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4354710280895233 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17003534734249115 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1197916641831398 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5304923057556152 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9795044660568237 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.492407888174057 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.973958432674408 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 _runtime‚4530.770651588 9"Purity/0FRt*SRqqt1FBK>çÍJt/event_purity‚0.6359511613845825 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38487643003463745 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5880626440048218 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.36913979053497314 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3821541368961334 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21007448434829712 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4611279368400574 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.38446399569511414 *jet/accuracy_2_of_5‚0.4715181291103363 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2545662224292755 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4131944477558136 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9891442656517029 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 particle/accuracy_3_of_7‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01816578581929207 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47333332896232605 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09900228679180145 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.47256943583488464 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44466549158096313 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.25058799982070923 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47165513038635254 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016843033954501152 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2732687294483185 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_6‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01872568018734455 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5553240180015564 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021802494302392006 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34153977036476135 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4956139922142029 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2300652414560318 particle/accuracy_1_of_6‚1 particle/accuracy_5_of_7‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1574983298778534 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3929910957813263 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00925220176577568 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42996862530708313 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4128113389015198 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.290640652179718 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.010062893852591515 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 *jet/accuracy_1_of_1‚0.4646221697330475 particle/accuracy_3_of_4‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04183848947286606 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9748655557632446 *jet/accuracy_1_of_4‚0.8998486399650574 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22047917544841766 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.27096235752105713 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3338955342769623 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4252488315105438 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3594643771648407 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5242854952812195 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.42626461386680603 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 jet/accuracy_1_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.27727001905441284 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4399225115776062 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35092592239379883 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473652839661 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4094456136226654 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45062679052352905 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9495954513549805 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23652885854244232 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2717472314834595 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt2FBt/SRqqt_purity‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5099207162857056 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.40120694041252136 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4162936806678772 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2797619104385376 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4088922441005707 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5376302599906921 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9950212240219116 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.018880076706409454 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.28320881724357605 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38556551933288574 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9840697050094604 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.36647650599479675 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04945584759116173 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19820554554462433 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5339694619178772 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.973611056804657 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8502809405326843 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.44878044724464417 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.040592044591903687 ,particle/sensitivity‚0.31357845664024353 epoch‚2 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5148489475250244 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5137388110160828 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973068237304688 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 +jet/accuracy_3_of_4‚0.06557126343250275 particle/accuracy_4_of_5‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5455648899078369 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.470631867647171 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4527410864830017 ,jet/accuracy_4_of_4‚0.005029045976698399 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.46800851821899414 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5467749238014221 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.44828805327415466 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 +jet/accuracy_2_of_3‚0.24178561568260193 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4439953863620758 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.375 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09434203058481216 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166567325592 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981160163879395 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14358574151992798 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0798070877790451 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19166401028633118 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3057471215724945 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 trainer/global_step‚3626 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.026672102510929108 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9904367923736572 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19177688658237457 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2578168213367462 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2708333432674408 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2545199990272522 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2471264749765396 jet/accuracy_4_of_5‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606059432029724 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2936026453971863 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_2‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.31257307529449463 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3471677899360657 _step‚296 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.25795233249664307 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.314414381980896 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.22839508950710297 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4508272707462311 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2637229561805725 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.24271857738494873 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3509804606437683 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19197367131710052 *jet/accuracy_1_of_3‚0.8572184443473816 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36000001430511475 jet/accuracy_2_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 particle/accuracy_1_of_4‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.35407450795173645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.27142226696014404 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172577261924744 (particle/f_score‚0.39531001448631287 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.44015243649482727 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26934024691581726 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4340665936470032 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 6 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.225917249917984 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0¨?9aLÌ‚(:„ ¸¶ëԡȬ™ gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.735' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.947 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âr)ÚÍ‚(:ö ¸¶ëÔ«ûÝ cpu‚25.341033803547923 network.sent‚ 8.602963e+06# disk.nvme0c0n1.in‚ 18957.0859375# disk.dm-0.out‚97672.71435546875 disk.dm-1.out‚ 143.11328125 disk.drbd1260.out‚ 0.70703125 disk.sda.in‚0! disk.nvme0n1.in‚ 18957.0859375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0# memory_percent‚93.5948133468628) proc.memory.availableMB‚ 1049.42578125" proc.memory.rssMB‚ 3037.5234375 proc.cpu.threads‚17 disk.sda3.in‚0( disk.nvme0c0n1.out‚97672.71435546875 disk.dm-0.in‚ 18957.0859375 disk.sda1.in‚0 disk.sda.out‚ 143.11328125 disk.sda2.in‚0& disk.nvme0n1.out‚97672.71435546875 network.recv‚ 1.523921e+06( disk./.usagePercent‚78.8904888961545% disk./.usageGB‚1175.5252952575684 disk.sda3.out‚ 143.11328125 disk.sda2.out‚0) proc.memory.percent‚18.53957176208496ÇÚ,Ä΂(:„ ǶëÔˆÛ–— gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.717 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÉKÙyÏ‚(:î ǶëÔ¤æå›# disk.nvme0c0n1.in‚ 18957.0859375 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 143.55078125! disk.nvme0n1.in‚ 18957.0859375$ memory_percent‚93.59478950500488" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 disk.sda1.in‚0 network.recv‚ 1.530738e+06 disk.sda3.out‚ 143.55078125& disk.nvme0c0n1.out‚98122.736328125 disk.dm-0.in‚ 18957.0859375( proc.memory.availableMB‚ 1049.4296875 proc.cpu.threads‚17$ disk./.usageGB‚1175.626132965088 disk.sda3.in‚0 disk.dm-1.out‚ 143.55078125$ disk.nvme0n1.out‚98122.736328125 cpu‚25.34026907782593 disk.sda1.out‚0 network.sent‚ 8.727223e+06( disk./.usagePercent‚78.8972561993146! disk.dm-0.out‚98122.736328125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.71484375 disk.sda2.in‚0 disk.sda2.out‚0OðÃ2Ð- _wandbruntime‚4561  _runtime‚4561yãFÇÑÁ trainer/global_step‚3649 % _timestamp‚1.7885335769470818e+09 _runtime‚4561.101082523 _step‚297 & lr-AdamW/pg1‚0.0007194125570370778 & lr-AdamW/pg2‚0.0007194125570370778æÔÌÒÆ trainer/global_step‚3649 % _timestamp‚1.7885335769470818e+09 _runtime‚4561.101082523 _step‚297 & lr-AdamW/pg1‚0.0007194125570370778 & lr-AdamW/pg2‚0.0007194125570370778©;‡ZI2Ó- _wandbruntime‚4561  _runtime‚4561Ôü§Û'ÔÕO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69623439 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149094199261184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99347735 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141830167319888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 $ _timestamp‚1.788533576949035e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156352466000896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99374079 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26449578 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99374079 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156352466000896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚149094199261184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚149094181258240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7258284742656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149068118309904 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7383357088256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚3649 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168997518 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156352717672448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69623437 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165285641603072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149094181258240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0™Ò»üÕÚO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41643918 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocateÙlGä&d‚ 7258518411264 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99374079 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26449578 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99374079 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156352466000896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚149094199261184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚149094181258240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7258284742656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149068118309904 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7383357088256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚3649 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168997518 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156352717672448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69623437 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165285641603072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149094181258240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69623439 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149094199261184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99347735 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141830167319888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 $ _timestamp‚1.788533576949035e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156352466000896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68110588 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156393967748608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69641712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7260248768512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169016139 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99374427 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 % _timestamp‚1.7885335785396047e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169042511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69641712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7239663458558 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _step‚299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99374427 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149107361135870 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69641714 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156393967748608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99400797 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674ƒžÃšë&ØåM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149133485298176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26455870 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169042511 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68111264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7260482450432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7385639076352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7239892439850 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156393716063744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41654720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68110588 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156393967748608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69641712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7260248768512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169016139 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99374427 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 % _timestamp‚1.7885335785396047e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169042511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69641712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7239663458558 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _step‚299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99374427 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149107361135870 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69641714 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156393967748608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99400797 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608«2®Ü2Ù- _wandbruntime‚4562  _runtime‚4562܆k\ÚÖ /loss/FRt1/assignment_loss‚3.669502522494342 /loss/FBt2/detection_loss‚2.5194681882858276 %loss/total_loss‚43.77381818681138 0loss/FRt3/assignment_loss‚1.0604188967692225 /loss/FBt4/assignment_loss‚0.391476712945346 0loss/SRqqt2/detection_loss‚2.601194299500564 0loss/FRt4/assignment_loss‚3.6925134518567253 3loss/FBt3/assignment_loss‚8.811665785821369e-05 2loss/SRqqt3/assignment_loss‚1.1192464411155216 2loss/SRqqt4/assignment_loss‚1.9817795231938362 /loss/FRt1/detection_loss‚2.1324909216648833 /loss/FRt3/detection_loss‚1.5500552863078683 epoch‚3 0loss/FRt2/assignment_loss‚3.2229153712590537 2loss/SRqqt1/assignment_loss‚1.5279090130720667 0loss/FBt1/assignment_loss‚0.3398177938291198 .loss/FRt2/detection_loss‚3.063143836127387 1loss/SRqqt3/detection_loss‚1.0765976850357322 /loss/FBt4/detection_loss‚1.3572495282604926 trainer/global_step‚3649 _runtime‚4562.695354545 1loss/FBt2/assignment_loss‚0.12724909535010434 /loss/FRt4/detection_loss‚2.6656372091349434 .loss/FBt1/detection_loss‚2.560272231698036 _step‚300 1loss/SRqqt2/assignment_loss‚1.846769577470319 0loss/SRqqt4/detection_loss‚2.760025820948861 /loss/FBt3/detection_loss‚0.6696849916225824 % _timestamp‚1.7885335785419316e+09 1loss/SRqqt1/detection_loss‚1.8383116722106934 îËZÛÛ 1loss/SRqqt2/assignment_loss‚1.846769577470319 0loss/SRqqt4/detection_loss‚2.760025820948861 /loss/FBt3/detection_loss‚0.6696849916225824 % _timestamp‚1.7885335785419316e+09 1loss/SRqqt1/detection_loss‚1.8383116722106934 /loss/FRt1/assignment_loss‚3.669502522494342 /loss/FBt2/detection_loss‚2.5194681882858276 %loss/total_loss‚43.77381818681138 0loss/FRt3/assignment_loss‚1.0604188967692225 /loss/FBt4/assignment_loss‚0.391476712945346 0loss/SRqqt2/detection_loss‚2.601194299500564 0loss/FRt4/assignment_loss‚3.6925134518567253 3loss/FBt3/assignment_loss‚8.811665785821369e-05 2loss/SRqqt3/assignment_loss‚1.1192464411155216 2loss/SRqqt4/assignment_loss‚1.9817795231938362 /loss/FRt1/detection_loss‚2.1324909216648833 /loss/FRt3/detection_loss‚1.5500552863078683 epoch‚3 0loss/FRt2/assignment_loss‚3.2229153712590537 2loss/SRqqt1/assignment_loss‚1.5279090130720667 0loss/FBt1/assignment_loss‚0.3398177938291198 .loss/FRt2/detection_loss‚3.063143836127387 1loss/SRqqt3/detection_loss‚1.0765976850357322 /loss/FBt4/detection_loss‚1.3572495282604926 trainer/global_step‚3649 _runtime‚4562.695354545 1loss/FBt2/assignment_loss‚0.12724909535010434 /loss/FRt4/detection_loss‚2.6656372091349434 .loss/FBt1/detection_loss‚2.560272231698036 _step‚3ÛK§00¬“x«ì‘Ü‚(:† Ö¶ëÔã ý– gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.9726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”ZBèÝ‚(:ô Ö¶ëԚáš) disk./.usagePercent‚78.91067585825195% disk./.usageGB‚1175.8260955810547$ disk.nvme0n1.out‚98731.099609375 disk.drbd1260.out‚ 0.71484375) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17" disk.nvme0n1.in‚19125.73046875 disk.dm-0.in‚19125.73046875 disk.drbd1260.in‚0$ memory_percent‚93.60129833221436) proc.memory.availableMB‚ 1048.36328125& disk.nvme0c0n1.out‚98731.099609375 disk.sda.in‚0 disk.sda.out‚ 143.91796875 disk.sda2.out‚0 disk.sda3.in‚0! disk.dm-0.out‚98731.099609375 disk.dm-1.out‚ 143.91796875 disk.sda1.in‚0 network.recv‚ 1.53401e+06 disk.sda2.in‚0 disk.sda3.out‚ 143.91796875$ disk.nvme0c0n1.in‚19125.73046875 disk.dm-1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3037.5234375 cpu‚25.339504672523454 network.sent‚ 8.732105e+06² 0ûœÞ‚(:‘ å¶ëÔíÚ‰˜ gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.952' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.790399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ôwß‚(:ì å¶ëÔ“Å­š( disk.nvme0c0n1.out‚99173.27978515625 disk.dm-0.in‚19125.73046875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 144.40625& disk.nvme0n1.out‚99173.27978515625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚0.71875 disk.sda2.in‚0( proc.memory.availableMB‚ 1048.6953125) proc.memory.percent‚18.53957176208496 network.sent‚ 8.840745e+06 network.recv‚ 1.540662e+06( disk./.usagePercent‚78.9122441604067 disk.dm-1.out‚ 144.40625 disk.sda.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚19125.73046875 disk.sda3.in‚0$ disk./.usageGB‚1175.849464416504# disk.dm-0.out‚99173.27978515625 disk.sda3.out‚ 144.40625$ memory_percent‚93.59927177429199" proc.memory.rssMB‚ 3037.5234375 cpu‚25.338835378773428 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚19125.73046875åN¸à‚(:… ô¶ëÔòñõ— gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.168' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6336 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F>ÛÝ€á‚(:õ ô¶ëÔïòœ network.sent‚ 8.845647e+06 network.recv‚ 1.543938e+06 disk.sda1.out‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 cpu‚25.338044533728116) disk./.usagePercent‚78.94244383317583 disk.sda2.in‚0 disk.sda3.out‚ 144.76171875$ disk.nvme0c0n1.in‚19125.73046875$ memory_percent‚93.60265731811523& disk.nvme0n1.out‚99440.81884765625$ disk./.usageGB‚1176.299461364746 disk.sda.in‚0 disk.sda.out‚ 144.76171875 disk.sda2.out‚0" disk.nvme0n1.in‚19125.73046875( disk.nvme0c0n1.out‚99440.81884765625 disk.dm-0.in‚19125.73046875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.71875 disk.sda3.in‚0# disk.dm-0.out‚99440.81884765625 disk.dm-1.out‚ 144.76171875' proc.memory.availableMB‚ 1048.140625 proc.cpu.threads‚17V¿„§‘â‚(:† ƒ·ëÔùŸö¤ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xæ½Ð}ã‚(:ò ƒ·ëÔðÿÉ® disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 1.548252e+06( disk./.usagePercent‚78.9424451132135 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.in‚19125.73046875% disk.nvme0n1.out‚99532.0908203125$ disk.nvme0c0n1.in‚19125.73046875% disk./.usageGB‚1176.2994804382324 disk.sda.in‚0 disk.sda.out‚ 145.07421875 disk.sda2.out‚0$ proc.memory.availableMB‚1047.875" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 cpu‚25.337444229739326 disk.dm-0.in‚19125.73046875" disk.dm-0.out‚99532.0908203125 disk.dm-1.out‚ 145.07421875 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 145.07421875' disk.nvme0c0n1.out‚99532.0908203125$ memory_percent‚93.60427856445312 network.sent‚ 8.853725e+06 disk.drbd1260.out‚ 0.72265625Å 2ä- _wandbruntime‚4626  _runtime‚4626¤’+ÇåÁ trainer/global_step‚3699 % _timestamp‚1.7885336424721532e+09 _runtime‚4626.625774089 _step‚301 & lr-AdamW/pg1‚0.0007155628482686235 & lr-AdamW/pg2‚0.0007155628482686235yt½ÌæÆ _runtime‚4626.625774089 _step‚301 & lr-AdamW/pg1‚0.0007155628482686235 & lr-AdamW/pg2‚0.0007155628482686235 trainer/global_step‚3699 % _timestamp‚1.7885336424721532e+09­䲓Y2ç- _wandbruntime‚4626  _runtime‚4626Å¢ß'èÙO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158415235228672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7356734117888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7335643010080 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167469478340096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171222874 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70537189 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42184018 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151042574221612 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167469792899584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151058501110784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159986160069120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100712031 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158414983557120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26797277 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143706702230448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158415235228672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42184016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚302 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143706685191088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7356500449280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68981994 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚4626.627412236 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100685687 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159986184009216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7335871991164 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26797976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70537187 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171249220 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7356500449280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7356734117888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70537187 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚3699 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7483318270976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151058483107840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151058483107840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273Byä'éÞO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70537187 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171249220 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7356500449280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7356734117888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70537187 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚3699 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7483318270976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151058483107840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151058483107840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158415235228672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7356734117888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7335643010080 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167469478340096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171222874 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70537189 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42184018 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151042574221612 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167469792899584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151058501110784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159986160069120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100712031 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158414983557120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26797277 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143706702230448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158415235228672 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42184016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚302 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143706685191088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7356500449280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68981994 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚4626.627412236 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100685687 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159986184009216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7335871991164 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26797976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273® ŸÙ2ê- _wandbruntime‚4628  _runtime‚4628¼Äÿ UëäM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42194820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151097787147776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7358464475136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70555464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151082063068230 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167513470087680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7358464475136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68999412 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160027867979264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143744232587872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151081817047578 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158456485304832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151097787147776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100738749 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69000107 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7485305210880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885336440468872e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70555462 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100712379 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7358698157056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚3699 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7485602108416~Û«•  SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70555464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7358698157056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171294213 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7337830480358 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70555462 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158456485304832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151097769144832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171267841 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26805287 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151097769144832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160027844039168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _runtime‚4628.201541638 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100712379 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171294213 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171294213 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7337830480358 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70555462 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158456485304832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151097769144832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171267841 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26805287 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151097769144832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160027844039168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _runtime‚4628.201541638 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100712379 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171294213 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151097787147776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7358464475136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70555464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151082063068230 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167513470087680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7358464475136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68999412 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160027867979264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143744232587872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151081817047578 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158456485304832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151097787147776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100738749 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69000107 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7485305210880 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885336440468872e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70555462 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100712379 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7358698157056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚3699 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7485602108416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70555464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7358698157056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7433581050588 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162070382511616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169653514969088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145583203062288 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102049983 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160477501113344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153017030133320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173474576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160477501113344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160477752784896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7433810031672 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160477752784896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚3749 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7583453308416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚4689.626168864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153022784957440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7454716155904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7454716155904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69874146 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71450937 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346ÕÛ½¯çýßO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160477501113344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153017030133320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytùmbþ%es.all.peak‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173474576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160477501113344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160477752784896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7433810031672 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160477752784896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚3749 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7583453308416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚4689.626168864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153022784957440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7454716155904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7454716155904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69874146 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71450937 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7454949824512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42724118 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153016784112876 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153022784957440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162070358571520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102023639 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7433581050588 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162070382511616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169653514969088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145583203062288 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102049983²*FË?2þ- _wandbruntime‚4690  _runtime‚4690ËÖé‚ê&ÿäM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169697189443584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42734918 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173519543 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42734920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162112042541568 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153062088997376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160519002861056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27157094 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7585443799552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153056518979938 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173519543 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7456913863680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 % _timestamp‚1.7885337066442442e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102050331 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7435768520866 trainer/global_step‚3749 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160519002861056 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71469214 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153062070994432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71469212 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153062088997376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102076701 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69891341 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71469212 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102050331 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173545915 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71469214 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153056272959286 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0i‘øï&€éM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102076701 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69891341 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71469212 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102050331 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173545915 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71469214 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153056272959286 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169697189443584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42734918 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173519543 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42734920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162112042541568 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153062088997376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160519002861056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27157094 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7585443799552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153056518979938 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173519543 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7456913863680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 % _timestamp‚1.7885337066442442e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102050331 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7435768520866 trainer/global_step‚3749 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160519002861056 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71469214 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153062070994432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71469212 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153062088997376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496³ZÆI¿2- _wandbruntime‚4690  _runtime‚4690Ê2ô]‚× /loss/FRt1/assignment_loss‚3.563965658346812 2loss/FBt3/assignment_loss‚0.001331158599707251 .loss/FRt1/detection_loss‚2.339400215790822 1loss/SRqqt2/detection_loss‚2.3341855471198625 .loss/FRt3/detection_loss‚1.633422987203892 trainer/global_step‚3749 1loss/SRqqt2/assignment_loss‚1.439021730745161 0loss/FBt1/assignment_loss‚0.2213531396078262 _runtime‚4690.800347028 &loss/total_loss‚44.093080967257805 % _timestamp‚1.7885337066469052e+09 0loss/FRt2/assignment_loss‚2.8444234354551448 0loss/FRt3/assignment_loss‚1.3819623426215288 1loss/SRqqt3/assignment_loss‚1.333801093387107 2loss/SRqqt4/assignment_loss‚2.0000127811950668 /loss/FRt4/detection_loss‚3.4694144785404206 /loss/FBt4/detection_loss‚1.5224562566126547 0loss/FBt2/assignment_loss‚0.3006828157212567 /loss/FBt1/detection_loss‚2.8736068844795226 epoch‚3 0loss/FRt4/assignment_loss‚3.2007947832345964 1loss/SRqqt1/detection_loss‚2.1848098369021165 0loss/SRqqt4/detection_loss‚2.553146247510557 1loss/FBt4/assignment_loss‚0.10124287409385693 /loss/FRt2/detection_loss‚2.6302135295646134 1loss/SRqqt3/detection_loss‚0.8925211292381088 /loss/FBt2/detection_loss‚2.9952523350715636 2loss/SRqqt1/assignment_loss‚1.4293695650994778 /loss/FBt3/detection_loss‚0.8466901411161278 _step‚308]_çbƒÜ _step‚308 /loss/FRt1/assignment_loss‚3.563965658346812 2loss/FBt3/assignment_loss‚0.001331158599707251 .loss/FRt1/detection_loss‚2.339400215790822 1loss/SRqqt2/detection_loss‚2.3341855471198625 .loss/FRt3/detection_loss‚1.633422987203892 trainer/global_step‚3749 1loss/SRqqt2/assignment_loss‚1.439021730745161 0loss/FBt1/assignment_loss‚0.2213531396078262 _runtime‚4690.800347028 &loss/total_loss‚44.093080967257805 % _timestamp‚1.7885337066469052e+09 0loss/FRt2/assignment_loss‚2.8444234354551448 0loss/FRt3/assignment_loss‚1.3819623426215288 1loss/SRqqt3/assignment_loss‚1.333801093387107 2loss/SRqqt4/assignment_loss‚2.0000127811950668 /loss/FRt4/detection_loss‚3.4694144785404206 /loss/FBt4/detection_loss‚1.5224562566126547 0loss/FBt2/assignment_loss‚0.3006828157212567 /loss/FBt1/detection_loss‚2.8736068844795226 epoch‚3 0loss/FRt4/assignment_loss‚3.2007947832345964 1loss/SRqqt1/detection_loss‚2.1848098369021165 0loss/SRqqt4/detection_loss‚2.553146247510557 1loss/FBt4/assignment_loss‚0.10124287409385693 /loss/FRt2/detection_loss‚2.6302135295646134 1loss/SRqqt3/detection_loss‚0.8925211292381088 /loss/FBt2/detection_loss‚2.9952523350715636 2loss/SRqqt1/assignment_loss‚1.4293695650994778 /loss/FBt3/detection_loss‚0.8466901411161278´ ÷ÚV΄‚(:… ηëÔ‰Ôë˜ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.265'£~¢ä gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.253 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õFv„…‚(:ù ηëÔÏÕÏš disk.sda2.in‚0" disk.nvme0n1.in‚19260.33984375 disk.sda3.out‚ 147.6953125) proc.memory.availableMB‚ 1047.23046875 network.sent‚ 8.981486e+06) disk.nvme0c0n1.out‚100864.04345703125 disk.sda1.in‚0 disk.sda.out‚ 147.6953125' disk.nvme0n1.out‚100864.04345703125 disk.sda3.in‚0# memory_percent‚93.6082124710083) proc.memory.percent‚18.53957176208496 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.73046875) disk./.usagePercent‚78.97698513771813$ disk.nvme0c0n1.in‚19260.33984375$ disk.dm-0.out‚100864.04345703125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3037.5234375 cpu‚25.333901498483232 proc.cpu.threads‚17 network.recv‚ 1.570156e+06$ disk./.usageGB‚1176.814151763916 disk.dm-0.in‚19260.33984375 disk.dm-1.out‚ 147.69531256ÊG7‘†‚(:† Ý·ëԸ刖 gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.472' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8944 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05'7‡‚(:ô Ý·ëÔ«†¢›# memory_percent‚93.6115026473999 proc.cpu.threads‚17) disk.nvme0c0n1.out‚101030.02685546875 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.73828125 disk.nvme0n1.in‚ 19402.890625" proc.memory.rssMB‚ 3037.5234375 disk.dm-0.in‚ 19402.890625 disk.sda.in‚0 disk.sda.out‚ 148.1953125 disk.sda2.out‚0' disk.nvme0n1.out‚101030.02685546875 disk.sda3.out‚ 148.1953125) proc.memory.percent‚18.53957176208496 network.recv‚ 1.577775e+06% disk./.usageGB‚1176.8120918273926" disk.nvme0c0n1.in‚ 19402.890625 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1046.69140625 cpu‚25.333102776802626 network.sent‚ 9.090708e+06) disk./.usagePercent‚78.97684689364964$ disk.dm-0.out‚101030.02685546875 disk.dm-1.out‚ 148.1953125 disk.sda1.out‚0 disk.sda3.in‚0ÎÒb:ˆ‚(:… ì·ëÔÂ’ì˜ gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1528 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤€‰‚(:õ ì·ëÔÔÔùš disk.nvme0n1.in‚ 19402.890625) disk.nvme0c0n1.out‚101183.47216796875 disk.dm-0.in‚ 19402.890625) disk./.usagePercent‚78.97684561361197' disk.nvme0n1.out‚101183.47216796875 disk.sda3.out‚ 148.6015625" disk.nvme0c0n1.in‚ 19402.890625 disk.dm-1.in‚0 disk.dm-1.out‚ 148.6015625$ memory_percent‚93.61002445220947) proc.memory.availableMB‚ 1046.93359375 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3037.5234375 proc.cpu.threads‚17$ disk.dm-0.out‚101183.47216796875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.73828125) proc.memory.percent‚18.53957176208496 cpu‚25.332477018063386 network.sent‚ 9.095656e+06 network.recv‚ 1.581012e+06% disk./.usageGB‚1176.8120727539062 disk.sda.out‚ 148.6015625 disk.sda2.in‚0$P*;‘Š‚(:† û·ëÔ¦«Æ– gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.662' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5324 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì,~¢x‹‚(:í û·ëÔ€š$ memory_percent‚93.61000061035156 cpu‚25.331704638735946 proc.cpu.threads‚17) disk./.usagePercent‚78.97684689364964 disk.dm-0.in‚ 19402.890625 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.in‚0% proc.memory.availableMB‚ 1046.9375" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 network.recv‚ 1.585582e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 149.1328125 disk.drbd1260.out‚ 0.7421875 disk.sda2.out‚0 disk.nvme0n1.in‚ 19402.890625& disk.nvme0n1.out‚101352.2763671875 disk.sda3.out‚ 149.1328125 network.sent‚ 9.101777e+06% disk./.usageGB‚1176.8120918273926" disk.nvme0c0n1.in‚ 19402.890625( disk.nvme0c0n1.out‚101352.2763671875# disk.dm-0.out‚101352.2763671875 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 149.1328125 disk.sda2.in‚0î~QL2Œ- _wandbruntime‚4750  _runtime‚4750ã®ÀÇÁ & lr-AdamW/pg1‚0.0007076721505349216 & lr-AdamW/pg2‚0.0007076721505349216 trainer/global_step‚3799 % _timestamp‚1.7885337665368633e+09 _runtime‚4750.690389364 _step‚309q†ÌŽÆ & lr-AdamW/pg1‚0.0007076721505349216 & lr-AdamW/pg2‚0.0007076721505349216 trainer/global_step‚3799 % _timestamp‚1.7885337665368633e+09 _runtime‚4750.690389364 _step‚309µš È2- _wandbruntime‚4750  _runtime‚4750w_BKß'ÙO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7553165531136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27496334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147459720933488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154987104809984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154987086807040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171837820948480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7553165531136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154987086807040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171837506388992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43264216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 $ _timestamp‚1.788533766538035e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154991486045028 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175726278 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162540018669568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚162540018669568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7552931862528 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103361591 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43264218 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27497027 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154991240024584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103387935 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103387935 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72364689 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164154557073920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7552931862528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175752624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7682949315072 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103361591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚3799 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154987104809984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7683239934464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70761245 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164154581014016 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7553165531136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154987086807040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171837506388992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43264216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 $ _timestamp‚1.788533766538035e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154991486045028 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175726278 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162540018669568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚162540018669568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7552931862528 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103361591 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43264218 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27497027 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154991240024584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103387935 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103387935 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72364689 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164154557073920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7552931862528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175752624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7682949315072 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103361591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚3799 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154987104809984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7683239934464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70761245 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164154581014016 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7685229074944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162581520417280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72382964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175771245 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7554895888384 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103388283 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7533706561374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70778746 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155030728870994 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162581268732416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164196264984064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175771245 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72382964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 _step‚311 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚155026390846976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚155026390846976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175797617 PDeviceStatsMonitor.on_train_batch_end/í×lµ allocation.all.allocated‚ 175797617 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7533477580082 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147497251290912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7684942663168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27503726 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171881494059008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171881183707136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147497268330272 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43275018 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155030974891646 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103388283 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70778064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72382962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624…åÃ4ï&”éM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162581520417280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72382964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175771245 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7554895888384 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103388283 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7533706561374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70778746 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155030728870994 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162581268732416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164196264984064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175771245 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72382964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 _step‚311 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚155026390846976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚155026390846976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175797617 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175797617 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7533477580082 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147497251290912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7684942663168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27503726 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171881494059008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171881183707136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147497268330272 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43275018 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155030974891646 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103388283 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70778064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72382962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7685229074944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0·Në1è2•- _wandbruntime‚4751  _runtime‚4751S°V^–Ø 2loss/SRqqt2/assignment_loss‚1.7545560385204024 0loss/SRqqt4/detection_loss‚2.484327899912993 /loss/FBt2/detection_loss‚2.8139965263280002 /loss/FBt4/detection_loss‚1.5369060938396761 _runtime‚4751.689883652 % _timestamp‚1.7885337675364132e+09 %loss/total_loss‚42.58414518369554 epoch‚3 0loss/FRt4/assignment_loss‚2.9128667423802037 1loss/SRqqt3/assignment_loss‚1.065700840232346 1loss/FBt1/assignment_loss‚0.15555743386933804 /loss/FRt1/detection_loss‚3.4763846938256866 /loss/FRt3/detection_loss‚1.3814615483545227 1loss/SRqqt1/detection_loss‚2.0997361099968352 /loss/FRt1/assignment_loss‚3.565060686182093 0loss/FRt3/assignment_loss‚1.0874913496735654 3loss/FBt3/assignment_loss‚1.920433659072901e-05 1loss/SRqqt3/detection_loss‚0.9733963373537157 1loss/FBt2/assignment_loss‚0.16477772306693209 _step‚312 /loss/FBt3/detection_loss‚0.6091020049615985 0loss/FRt2/assignment_loss‚3.2685111115376153 /loss/FRt4/detection_loss‚2.9487704750030272 /loss/SRqqt2/detection_loss‚2.17775559425354 /loss/FBt1/detection_loss‚2.3908495207627616 2loss/SRqqt1/assignment_loss‚1.3269700385862961 2loss/SRqqt4/assignment_loss‚1.3766806482647855 /loss/FBt4/assignment_loss‚0.297456826466098 /loss/FRt2/detection_loss‚2.7158097359869213 trainer/global_step‚3799—Éác—Ý /loss/FBt3/detection_loss‚0.6091020049615985 0loss/FRt2/assignment_loss‚3.2685111115376153 /loss/FRt4/detection_loss‚2.9487704750030272 /loss/SRqqt2/detection_loss‚2.17775559425354 /loss/FBt1/detection_loss‚2.3908495207627616 2loss/SRqqt1/assignment_loss‚1.3269700385862961 2loss/SRqqt4/assignment_loss‚1.3766806482647855 /loss/FBt4/assignment_loss‚0.297456826466098 /loss/FRt2/detection_loss‚2.7158097359869213 trainer/global_step‚3799 2loss/SRqqt2/assignment_loss‚1.7545560385204024 0loss/SRqqt4/detection_loss‚2.484327899912993 /loss/FBt2/detection_loss‚2.8139965263280002 /loss/FBt4/detection_loss‚1.5369060938396761 _runtime‚4751.689883652 % _timestamp‚1.7885337675364132e+09 %loss/total_loss‚42.58414518369554 epoch‚3 0loss/FRt4/assignment_loss‚2.9128667423802037 1loss/SRqqt3/assignment_loss‚1.065700840232346 1loss/FBt1/assignment_loss‚0.15555743386933804 /loss/FRt1/detection_loss‚3.4763846938256866 /loss/FRt3/detection_loss‚1.3814615483545227 1loss/SRqqt1/detection_loss‚2.0997361099968352 /loss/FRt1/assignment_loss‚3.565060686182093 0loss/FRt3/assignment_loss‚1.0874913496735654 3loss/FBt3/assignment_loss‚1.920433659072901e-05 1loss/SRqqt3/detection_loss‚0.9733963373537157 1loss/FBt2/assignment_loss‚0.16477772306693209 _step‚312¸pÇ—°œ˜‚(:‘ Š¸ëÔ«ëæ˜ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.308' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.061600000000002 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎŒœ{™‚(:𠊸ëÔÏ‘ä) disk./.usagePercent‚78.97684561361197% disk./.usageGB‚1176.8120727539062" disk.nvme0c0n1.in‚ 19402.890625# disk.dm-0.out‚101697.9404296875 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 149.55078125 disk.dm-0.in‚ 19402.890625 disk.dm-1.out‚ 149.55078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 19402.890625 disk.sda3.in‚0 disk.sda3.out‚ 149.55078125) proc.memory.percent‚18.53957176208496 disk.drbd1260.out‚ 0.7421875( disk.nvme0c0n1.out‚101697.9404296875 disk.sda1.in‚0$ memory_percent‚93.61023902893066( proc.memory.availableMB‚ 1046.8984375" proc.memory.rssMB‚ 3037.5234375 proc.cpu.threads‚17 network.recv‚ 1.58886e+06 disk.dm-1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚101697.9404296875 cpu‚25.33104237707441 network.sent‚ 9.10661e+06&~‘š‚(:† ™¸ëÔÙ‰ù– gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^nG#‚›‚(:÷ ™¸ëÔƒ«Úš network.recv‚ 1.595992e+06) disk./.usagePercent‚78.97527065526133% disk./.usageGB‚1176.7886047363281 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 150.0078125# disk.dm-0.out‚101953.1005859375 disk.sda1.out‚0# memory_percent‚93.6103105545044) proc.memory.availableMB‚ 1046.88671875" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17 network.sent‚ 9.215611e+06 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.74609375 disk.sda.out‚ 150.0078125" disk.nvme0n1.in‚19539.41015625& disk.nvme0n1.out‚101953.1005859375 cpu‚25.330303098550587$ disk.nvme0c0n1.in‚19539.41015625( disk.nvme0c0n1.out‚101953.1005859375 disk.dm-0.in‚19539.41015625 disk.dm-1.out‚ 150.0078125 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0î\ñ᜜‚(:‘ ¨¸ëÔ·öâ” gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.308' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.061600000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(¬åp„‚(:ù ¨¸ëÔ¦Þš˜ disk.sda1.in‚0 network.sent‚ 9.220453e+06 network.recv‚ 1.598951e+06) disk./.usagePercent‚78.99361922725538$ disk.nvme0c0n1.in‚19539.41015625 disk.dm-0.in‚19539.41015625 disk.dm-1.in‚0 disk.sda2.out‚0# disk.dm-0.out‚102384.5732421875 disk.drbd1260.in‚0" disk.nvme0n1.in‚19539.41015625 disk.sda3.in‚0) proc.memory.availableMB‚ 1046.63671875 cpu‚25.329665300829948 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1260.out‚ 0.74609375( disk.nvme0c0n1.out‚102384.5732421875 disk.dm-1.out‚ 150.81640625 disk.sda.out‚ 150.81640625& disk.nvme0n1.out‚102384.5732421875# disk./.usageGB‚1177.06201171875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 150.81640625$ memory_percent‚93.61183643341064" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 û_‘ž‚(:† ·¸ëÔú¬Ì– gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.446' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0892 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò´óˆ{Ÿ‚(:ð ·¸ëÔÃñÆš disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚1177.061653137207" disk.nvme0n1.in‚19539.41015625 disk.sda3.out‚ 151.70703125 disk.dm-0.in‚19539.41015625 disk.dm-1.in‚0$ memory_percent‚94.65904235839844 network.sent‚ 9.226577e+06) disk.nvme0c0n1.out‚102544.41259765625 disk.drbd1260.out‚0.75 disk.sda.out‚ 151.70703125$ proc.memory.availableMB‚875.0625 cpu‚25.3289272433525 network.recv‚ 1.603614e+06$ disk.nvme0c0n1.in‚19539.41015625$ disk.dm-0.out‚102544.41259765625 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496) disk./.usagePercent‚78.99359516254717' disk.nvme0n1.out‚102544.41259765625 disk.sda3.in‚0 disk.dm-1.out‚ 151.70703125‰b¹ñ2 - _wandbruntime‚4809  _runtime‚4809˜|½Ç¡Á _step‚313 & lr-AdamW/pg1‚0.0007036328059792547 & lr-AdamW/pg2‚0.0007036328059792547 trainer/global_step‚3849 % _timestamp‚1.7885338257978957e+09 _runtime‚4809.951383291OÔmÌ¢Æ trainer/global_step‚3849 % _timestamp‚1.7885338257978957e+09 _runtime‚4809.951383291 _step‚313 & lr-AdamW/pg1‚0.0007036328059792547 & lr-AdamW/pg2‚0.0007036328059792547¹ý »2£- _wandbruntime‚4809  _runtime‚48099×åß'¤ÙO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7782791900160 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73278437 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71648714 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7651147569152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73278439 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166238779516416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73278439 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚3849 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104699543 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7651381237760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156951406659584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7651147569152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7629686112688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7629457131604 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156965941956736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104725887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43804316 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27843698 _runtime‚4809.952312692 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104699543 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156951388656640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156951388656640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174021855744512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174021547476480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 $ _timestamp‚1.788533825798675e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71648014 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149336255844048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156951406659584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164602536225792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164602787897344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43804318 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166238755576320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177977980 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73278437 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164602536225792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚314 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552–”…?¸¥ÞO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104699543 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156951388656640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156951388656640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174021855744512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174021547476480 LFDeviceStatsOñ 8,%Monitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 $ _timestamp‚1.788533825798675e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71648014 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149336255844048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156951406659584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164602536225792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164602787897344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43804318 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166238755576320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177977980 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73278437 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164602536225792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚314 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7782791900160 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73278437 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71648714 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7651147569152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73278439 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166238779516416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73278439 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚3849 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104699543 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7651381237760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156951406659584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7651147569152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7629686112688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7629457131604 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156965941956736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104725887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43804316 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27843698 _runtime‚4809.952312692º>`Jq2¦- _wandbruntime‚4811  _runtime‚4811Ü4²|ê&§äM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚315 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157005430803354 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156990692696576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73296712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73296714 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164644037973504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885338270054498e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71665560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7785076509696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7631644601882 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104726235 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7653111595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43815120 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4811.159292051 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71666240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156990674693632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166280463486464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157005184782702 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7653111595008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27851120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164644037973504 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104752605 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178022947 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104752605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7784796389376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149373786201472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149373769162112 trainer/global_step‚3849 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73296712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174065235935744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7631415620590 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73296714 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178049319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178022947 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174065539996160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156990692696576 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104726235cTÛ·ï&¨éM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7631644601882 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104726235 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7653111595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43815120 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4811.159292051 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71666240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156990674693632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166280463486464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157005184782702 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7653111595008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27851120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164644037973504 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104752605 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178022947 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104752605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7784796389376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149373786201472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149373769162112 trainer/global_step‚3849 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73296712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174065235935744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7631415620590 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73296714 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178049319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178022947 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174065539996160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156990692696576 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104726235 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚315 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157005430803354 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156990692696576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73296712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73296714 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164644037973504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885338270054498e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71665560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7785076509696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1»ÛÌÇÊ2©- _wandbruntime‚4811  _runtime‚4811{f\ªÖ 0loss/FRt4/assignment_loss‚3.3558714985847473 2loss/SRqqt1/assignment_loss‚1.7238196255475806 /loss/FRt3/detection_loss‚1.4986253335815052 /loss/FBt3/detection_loss‚0.8884438054519705 _step‚316 /loss/FRt2/detection_loss‚2.9627002734562446 %loss/total_loss‚44.49644163304393 trainer/global_step‚3849 0loss/FRt1/assignment_loss‚3.8203000882092644 1loss/SRqqt3/detection_loss‚1.0365314635909297 /loss/FBt1/detection_loss‚2.7048821540979238 _runtime‚4811.161145615 0loss/FRt2/assignment_loss‚3.0690213462402083 /loss/FRt4/detection_loss‚2.9333769977092743 0loss/SRqqt2/detection_loss‚2.299284818768501 % _timestamp‚1.7885338270077727e+09 0loss/FBt1/assignment_loss‚0.5175479469991597 0loss/FBt4/assignment_loss‚0.2691509706631546 /loss/SRqqt1/detection_loss‚1.89876869425439 /loss/FBt2/detection_loss‚2.2315087504684925 epoch‚3 /loss/FRt3/assignment_loss‚1.689966092164489 1loss/SRqqt4/assignment_loss‚1.749575703906325 3loss/FBt3/assignment_loss‚7.754508998814025e-05 0loss/SRqqt4/detection_loss‚2.377046878521259 2loss/SRqqt2/assignment_loss‚2.1042044021189215 2loss/SRqqt3/assignment_loss‚1.1146658044127584 0loss/FBt2/assignment_loss‚0.3714799893423333 /loss/FBt4/detection_loss‚1.3294904922184192 /loss/FRt1/detection_loss‚2.5501009576460936^õÍa«Û /loss/FRt1/detection_loss‚2.5501009576460936 0loss/FRt4/assignment_loss‚3.3558714985847473 2loss/SRqqt1/assignment_loss‚1.7238196255475806 /loss/FRt3/detection_loss‚1.4986253335815052 /loss/FBt3/detection_loss‚0.8884438054519705 _step‚316 /loss/FRt2/detection_loss‚2.9627002734562446 %loss/total_loss‚44.49644163304393 trainer/global_step‚3849 0loss/FRt1/assignment_loss‚3.8203000882092644 1loss/SRqqt3/detection_loss‚1.0365314635909297 /loss/FBt1/detection_loss‚2.7048821540979238 _runtime‚4811.161145615 0loss/FRt2/assignment_loss‚3.0690213462402083 /loss/FRt4/detection_loss‚2.9333769977092743 0loss/SRqqt2/detection_loss‚2.299284818768501 % _timestamp‚1.7885338270077727e+09 0loss/FBt1/assignment_loss‚0.5175479469991597 0loss/FBt4/assignment_loss‚0.2691509706631546 /loss/SRqqt1/detection_loss‚1.89876869425439 /loss/FBt2/detection_loss‚2.2315087504684925 epoch‚3 /loss/FRt3/assignment_loss‚1.689966092164489 1loss/SRqqt4/assignment_loss‚1.749575703906325 3loss/FBt3/assignment_loss‚7.754508998814025e-05 0loss/SRqqt4/detection_loss‚2.377046878521259 2loss/SRqqt2/assignment_loss‚2.1042044021189215 2loss/SRqqt3/assignment_loss‚1.1146658044127584 0loss/FBt2/assignment_loss‚0.3714799893423333 /loss/FBt4/detection_loss‚1.3294904922184192¼ØV«m›¬‚(: ƸëÔ꿽˜ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.71' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.541999999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡Ž <ï í6u­‚(:ê ƸëÔâ˜÷™ disk.sda1.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚19539.41015625 disk.sda3.out‚152.3125# memory_percent‚94.6547269821167) disk./.usagePercent‚79.00893078187077% disk./.usageGB‚1177.2901649475098 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 875.76953125" proc.memory.rssMB‚ 3037.5234375) disk.nvme0c0n1.out‚102635.49072265625 disk.drbd1260.out‚0.75 disk.sda.out‚152.3125 proc.cpu.threads‚17 network.recv‚ 1.607006e+06 disk.dm-0.in‚19539.41015625$ disk.dm-0.out‚102635.49072265625 disk.sda.in‚0' disk.nvme0n1.out‚102635.49072265625 disk.sda3.in‚0) proc.memory.percent‚18.53957176208496 cpu‚25.328174616865386 network.sent‚ 9.231469e+06$ disk.nvme0c0n1.in‚19539.41015625 disk.dm-1.out‚152.3125¯8qe‘®‚(:† Õ¸ëÔåÖ²˜ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5436 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡rf¯‚(:ö Õ¸ëÔê¬Óœ% disk./.usageGB‚1177.2891273498535 disk.sda2.in‚0& disk.nvme0n1.out‚103118.0693359375$ disk.nvme0c0n1.in‚19685.58984375 disk.dm-1.out‚ 152.7421875 disk.drbd1260.out‚ 0.75390625( proc.memory.availableMB‚ 875.23828125 disk.sda2.out‚0 disk.dm-0.in‚19685.58984375# disk.dm-0.out‚103118.0693359375 disk.drbd1260.in‚0$ memory_percent‚94.65796947479248) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17 network.sent‚ 9.34026e+06) disk./.usagePercent‚79.00886114782145 disk.sda3.in‚0 disk.sda3.out‚ 152.7421875 cpu‚25.327493192844962 network.recv‚ 1.613688e+06 disk.sda.in‚0 disk.sda.out‚ 152.7421875" disk.nvme0n1.in‚19685.58984375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3037.5234375( disk.nvme0c0n1.out‚103118.0693359375Ë®ßT‘°‚(:† ä¸ëÔù£¬— gpu.0.gpu‚10.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5952 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö—a«x±‚(:í ä¸ëÔ“ÆÓš network.sent‚ 9.344325e+06) disk./.usagePercent‚79.00886268386665 disk.sda.in‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚103349.0576171875 disk.dm-1.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 disk.sda.out‚153.5625 disk.sda2.in‚0& disk.nvme0n1.out‚103349.0576171875$ disk.nvme0c0n1.in‚19685.58984375 disk.dm-1.out‚153.5625 disk.sda1.out‚0 network.recv‚ 1.616175e+06$ disk./.usageGB‚1177.289150238037 disk.sda2.out‚0 disk.sda3.out‚153.5625 disk.dm-0.in‚19685.58984375$ memory_percent‚94.65644359588623( proc.memory.availableMB‚ 875.48828125" proc.memory.rssMB‚ 3037.5234375" disk.nvme0n1.in‚19685.58984375# disk.dm-0.out‚103349.0576171875 disk.sda1.in‚0 disk.drbd1260.out‚ 0.75390625) proc.memory.percent‚18.53957176208496 cpu‚25.326815957929885Q|粂(:… ó¸ëÔÓ÷• gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7826 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t2+Wt³‚(:é ó¸ëÔÛõ´š disk.dm-0.in‚19685.58984375" disk.dm-0.out‚103732.630859375 disk.dm-1.out‚ 154.21875& proc.memory.availableMB‚ 874.984375 cpu‚25.326021635992472) disk./.usagePercent‚79.00886268386665 disk.sda2.out‚0% disk.nvme0n1.out‚103732.630859375 disk.sda3.in‚0 disk.sda3.out‚ 154.21875' disk.nvme0c0n1.out‚103732.630859375 disk.sda1.out‚0) proc.memory.percent‚18.53957176208496 network.recv‚ 1.62073e+06 disk.sda2.in‚0" disk.nvme0n1.in‚19685.58984375 disk.dm-1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚1177.289150238037 disk.sda.in‚0$ disk.nvme0c0n1.in‚19685.58984375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7578125$ memory_percent‚94.65951919555664" proc.memory.rssMB‚ 3037.5234375 network.sent‚ 9.350549e+06 disk.sda.out‚ 154.21875[ÌÒg2´- _wandbruntime‚4867  _runtime‚4867ÚâñƵÀ & lr-AdamW/pg1‚0.0006995319156883563 & lr-AdamW/pg2‚0.0006995319156883563 trainer/global_step‚3899 % _timestamp‚1.7885338829922082e+09 _runtime‚ 4867.14601386 _step‚317eã;‰Ë¶Å _runtime‚ 4867.14601386 _step‚317 & lr-AdamW/pg1‚0.0006995319156883563 & lr-AdamW/pg2‚0.0006995319156883563 trainer/global_step‚3899 % _timestamp‚1.7885338829922082e+09½/»K-2·- _wandbruntime‚4867  _runtime‚4867x”­và'¸ÚO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72535961 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176205944339968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168322954078720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7727395172112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74192187 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7882671507456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180256028 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158915708509184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7749363275776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44344416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168322978018816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚166665053782016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74192189 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151212773715248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28190846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72535262 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚4867.147683026 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106063839 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166665305453568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166665053782016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106037495 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7749363275776 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944GL>å'¹ßO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚318 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44344418 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180229682 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚166665053782016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74192189 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151212773715248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28190846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72535262 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚4867.147683026 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106063839 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166665305453568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166665053782016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106037495 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7749363275776 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72535961 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176205944339968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168322954078720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7727395172112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74192187 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7882671507456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180256028 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158915708509184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7749363275776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44344416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168322978018816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168364638048768 _runtime‚4868.573108626 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166706555529728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151250287033312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 325031936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7729353661098 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180274649 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176249621921280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106090557 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 301091840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7751327301632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74210464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180301021 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72553504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7751560983552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7884959932416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158954976543232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106090557 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158954994546176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158979886715062 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151250304072672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚319 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166706303844864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7751560983552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚3899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166706555529728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176249296889344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74210462 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180274649 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74210462 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158979640694410 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72552797 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44355218 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7729582642390 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2˜íʸï&¼éM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180274649 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176249621921280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106090557 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 301091840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7751327301632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74210464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180301021 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72553504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7751560983552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7884959932416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158954976543232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106090557 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158954994546176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158979886715062 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151250304072672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚319 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166706303844864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7751560983552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚3899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166706555529728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176249296889344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74210462 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180274649 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74210462 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158979640694410 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72552797 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44355218 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7729582642390 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180301021 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106064187 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158954994546176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7751327301632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168364638048768 _runtime‚4868.573108626 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166706555529728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151250287033312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 325031936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7729353661098 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7847578982400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28537551 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160914853780152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168727823009792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160914607759708 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107401791 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚178389610763776 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168727571338240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75105939 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885339412242153e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 trainer/global_step‚3949 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44884516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75105937 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75105937 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153089291586448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107375447 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7825562193704 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182481384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7982458182656 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168727823009792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73422067 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44884518 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160880010358784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168727571338240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160880010358784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7847812651008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153089274547088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 Ã†I±ÍÞO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160914607759708 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107401791 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚178389610763776 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_staÝ$vS3$rt/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168727571338240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75105939 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885339412242153e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 trainer/global_step‚3949 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44884516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75105937 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75105937 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153089291586448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107375447 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7825562193704 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182481384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7982458182656 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168727823009792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73422067 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44884518 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160880010358784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168727571338240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160880010358784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7847812651008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153089274547088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚ 4925.37806061 _step‚322 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28538243 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160879992355840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7982744607744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7847578982400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28537551 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160914853780152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168727823009792ÂK$6 2Î- _wandbruntime‚4926  _runtime‚4926áˆþÄê&ÏäM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178433603159552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7984742668288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160919278392832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107402139 _runtime‚4926.634717463 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75124214 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7885339424798594e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160954096606118 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73439699 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168768821401088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75124214 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28545087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28544381 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182552723 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73440407 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7827291701606 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182526351 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160919278392832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182552723 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7849776690176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75124212 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7849543008256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7984449965056 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚323 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107428509 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160919296395776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44895318 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7827520682898 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168769073085952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160954342626770 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107428509 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚708 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44895320œÊ(Üï&ÐéM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7827520682898 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168769073085952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160954342626770 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107428509 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚708 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44895320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178433603159552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7984742668288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160919278392832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107402139 _runtime‚4926.634717463 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75124214 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7885339424798594e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160954096606118 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73439699 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168768821401088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75124214 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28545087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28544381 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182552723 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73440407 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7827291701606 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182526351 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160919278392832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182552723 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7849776690176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75124212 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7849543008256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7984449965056 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚323 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107428509 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160919296395776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44895318 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104ÃÓT/2Ñ- _wandbruntime‚4926  _runtime‚4926ûáó\ÒÖ 0loss/SRqqt1/detection_loss‚1.975902507454157 1loss/SRqqt4/detection_loss‚2.6451742931416162 epoch‚3 /loss/FRt3/assignment_loss‚1.670154572384698 1loss/SRqqt3/detection_loss‚1.0025616469793022 .loss/FBt2/detection_loss‚2.793552959958712 /loss/FBt3/detection_loss‚0.7744293114056393 /loss/FRt4/assignment_loss‚3.132536699374517 2loss/SRqqt2/assignment_loss‚1.7275999338867574 1loss/SRqqt3/assignment_loss‚1.061407474257673 _runtime‚4926.637037656 0loss/FRt2/assignment_loss‚2.8587647347614684 1loss/FBt4/assignment_loss‚0.11365126237093146 /loss/FRt4/detection_loss‚3.0237383395433426 % _timestamp‚1.7885339424834602e+09 .loss/FRt1/detection_loss‚2.410657738645875 0loss/FRt1/assignment_loss‚3.5047987782292895 0loss/FBt1/assignment_loss‚0.3345307515900913 trainer/global_step‚3949 _step‚324 1loss/SRqqt1/assignment_loss‚1.580520029552281 1loss/SRqqt4/assignment_loss‚2.043663177835314 1loss/FBt2/assignment_loss‚0.20873611139055961 /loss/FRt2/detection_loss‚2.7744216302345537 .loss/FBt4/detection_loss‚1.255629325058402 %loss/total_loss‚44.09125786104657 /loss/FRt3/detection_loss‚1.6062566361789192 1loss/SRqqt2/detection_loss‚2.4805218336673884 /loss/FBt1/detection_loss‚3.1120469599962233 4loss/FBt3/assignment_loss‚1.1531488562944068e-06D€paaÓÛ 1loss/SRqqt2/detection_loss‚2.4805218336673884 /loss/FBt1/detection_loss‚3.1120469599962233 4loss/FBt3/assignment_loss‚1.1531488562944068e-06 0loss/SRqqt1/detection_loss‚1.975902507454157 1loss/SRqqt4/detection_loss‚2.6451742931416162 epoch‚3 /loss/FRt3/assignment_loss‚1.670154572384698 1loss/SRqqt3/detection_loss‚1.0025616469793022 .loss/FBt2/detection_loss‚2.793552959958712 /loss/FBt3/detection_loss‚0.7744293114056393 /loss/FRt4/assignment_loss‚3.132536699374517 2loss/SRqqt2/assignment_loss‚1.7275999338867574 1loss/SRqqt3/assignment_loss‚1.061407474257673 _runtime‚4926.637037656 0loss/FRt2/assignment_loss‚2.8587647347614684 1loss/FBt4/assignment_loss‚0.11365126237093146 /loss/FRt4/detection_loss‚3.0237383395433426 % _timestamp‚1.7885339424834602e+09 .loss/FRt1/detection_loss‚2.410657738645875 0loss/FRt1/assignment_loss‚3.5047987782292895 0loss/FBt1/assignment_loss‚0.3345307515900913 trainer/global_step‚3949 _step‚324 1loss/SRqqt1/assignment_loss‚1.580520029552281 1loss/SRqqt4/assignment_loss‚2.043663177835314 1loss/FBt2/assignment_loss‚0.20873611139055961 /loss/FRt2/detection_loss‚2.7744216302345537 .loss/FBt4/detection_loss‚1.255629325058402 %loss/total_loss‚44.09125786104657 /loss/FRt3/detection_loss‚1.6062566361789192ÄT8Q‘Ô‚(:† ¾¹ëÔ…ö¹— gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0932 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊD/²Õ‚(:î ¾¹ëÔ›¸Ö™& disk.nvme0c0n1.out‚105303.97265625 disk.dm-0.in‚ 19896.1953125 disk.drbd1260.out‚0.765625$ memory_percent‚94.65422630310059' proc.memory.availableMB‚ 875.8515625 network.sent‚ 9.493061e+06# disk.nvme0c0n1.iL}ëÊvn‚ 19896.1953125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 156.85546875$ disk.nvme0n1.out‚105303.97265625 disk.sda3.out‚ 156.85546875% disk./.usageGB‚1177.6315307617188 disk.dm-1.out‚ 156.85546875 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 19896.1953125 cpu‚25.322719582811903 proc.cpu.threads‚17 network.recv‚ 1.642155e+06) disk./.usagePercent‚79.03184012808732! disk.dm-0.out‚105303.97265625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496ÑÎ5Ö‚(:… ͹ëÔŽÝ¥– gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.684' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3368 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02®Ozׂ(:ï ͹ëÔÝöø™% disk./.usageGB‚1177.6315307617188! disk.nvme0n1.in‚ 19896.1953125 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 19896.1953125" disk.dm-0.out‚105496.974609375 disk.sda1.out‚0$ memory_percent‚94.65441703796387) proc.memory.percent‚18.53957176208496) disk./.usagePercent‚79.03184012808732 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚105496.974609375' disk.nvme0c0n1.out‚105496.974609375 disk.dm-1.in‚0 network.sent‚ 9.589015e+06 disk.dm-0.in‚ 19896.1953125 disk.dm-1.out‚ 157.5390625' proc.memory.availableMB‚ 875.8203125" proc.memory.rssMB‚ 3037.5234375 network.recv‚ 1.649023e+06 disk.drbd1260.out‚ 0.76953125 disk.sda.out‚ 157.5390625 disk.sda3.out‚ 157.5390625 disk.sda1.in‚0 cpu‚25.32204688386657 proc.cpu.threads‚17¸ˆØ‚(:… ܹëÔ¹Òô– gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.436' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.6872 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pq¶‚Ù‚(:÷ ܹëÔ—êþ› network.sent‚ 9.595814e+06 disk.sda.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚20083.66015625' disk.nvme0c0n1.out‚105966.912109375) proc.memory.percent‚18.53957176208496 disk.drbd1260.out‚ 0.76953125 disk.sda.out‚ 158.01953125 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 158.01953125" disk.dm-0.out‚105966.912109375 disk.dm-1.out‚ 158.01953125" proc.memory.rssMB‚ 3037.5234375 network.recv‚ 1.652643e+06% disk./.usageGB‚1177.6315307617188 disk.sda1.in‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚105966.912109375$ disk.nvme0c0n1.in‚20083.66015625 disk.dm-1.in‚0( proc.memory.availableMB‚ 875.56640625) disk./.usagePercent‚79.03184012808732 disk.sda1.out‚0 disk.dm-0.in‚20083.66015625$ memory_percent‚94.65596675872803 cpu‚25.321509516288764 proc.cpu.threads‚17à#ŒœÚ‚(:‘ ë¹ëÔšˆì— gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.519' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.503800000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íûëÛ‚(:ö ë¹ëÔü‘­™ disk.dm-0.in‚20083.66015625$ memory_percent‚94.65601444244385) proc.memory.percent‚18.53957176208496 disk.sda3.out‚ 158.4140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7734375 proc.cpu.threads‚17 network.recv‚ 1.657086e+06 disk.sda.in‚0$ disk.dm-0.out‚106101.94287109375 disk.dm-1.in‚0 disk.dm-1.out‚ 158.4140625 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3037.5234375 network.sent‚ 9.600063e+06 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚20083.66015625 disk.sda3.in‚0) disk.nvme0c0n1.out‚106101.94287109375( proc.memory.availableMB‚ 875.55859375 cpu‚25.320774901910568' disk./.usagePercent‚79.031841408125$ disk./.usageGB‚1177.631549835205 disk.sda.out‚ 158.4140625' disk.nvme0n1.out‚106101.94287109375$ disk.nvme0c0n1.in‚20083.66015625 \1\2Ü- _wandbruntime‚4985  _runtime‚4985¥qìÇÝÁ _step‚325 & lr-AdamW/pg1‚0.0006911489290263624 & lr-AdamW/pg2‚0.0006911489290263624 trainer/global_step‚3999 % _timestamp‚1.7885340016907954e+09 _runtime‚4985.844459858}<½ÝÌÞÆ & lr-AdamW/pg1‚0.0006911489290263624 & lr-AdamW/pg2‚0.0006911489290263624 trainer/global_step‚3999 % _timestamp‚1.7885340016907954e+09 _runtime‚4985.844459858 _step‚325ÅÔ+¨2ß- _wandbruntime‚4985  _runtime‚4985ôÕ'Yß'àÙO % _timestamp‚1.7885340016923745e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184759432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108713399 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180574072464896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7923500234212 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170790088894464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162844294205440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180573772585472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108713399 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184733086 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7946028357632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76019687 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170790340566016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28885576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28884877 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162844294205440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154965792418288 trainer/global_step‚3999 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45424618 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45424616 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162889309691860 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162844312208384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170790088894464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108739743 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74309493 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162844312208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162889063671416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172491351083520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76019689 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76019687 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154965809457648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚326 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172491375023616l4iéä'áÞO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184759432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108713399 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180574072464896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7923500234212 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170790088894464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162844294205440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180573772585472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108713399 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184733086 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7946028357632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76019687 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170790340566016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28885576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28884877 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162844294205440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154965792418288 trainer/global_step‚3999 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45424618 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45424616 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162889309691860 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162844312208384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170790088894464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108739743 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74309493 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162844312208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162889063671416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172491351083520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76019689 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76019687 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154965809457648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚326 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172491375023616 % _timestamp‚1.7885340016923745e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309Æ0ª•>2â- _wandbruntime‚4987  _runtime‚4987µr}¡üãäM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162928552517826 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108740091 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180617450899968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8084704355328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162883580242432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7885340029226837e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184778053 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170831338957312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7947758714880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184778053 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45435418 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180617763348992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚4987.076935671 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚327 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108766461 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170831590642176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7947992396800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76037964 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74327949 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184804425 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162883580242432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155003322775712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3999 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45435420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8084415846400 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pT© Iî ool.allocated‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7947992396800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172533035053568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74327269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76037962 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108766461 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170831590642176 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162883598245376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162928798538478 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184804425 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76037962 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76037964 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74327949 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184804425 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162883580242432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155003322775712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3999 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45435420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8084415846400 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7947992396800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172533035053568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74327269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76037962 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108766461 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170831590642176 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162883598245376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162928798538478 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184804425 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76037962 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180617450899968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8084704355328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162883580242432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7885340029226837e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184778053 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170831338957312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7947758714880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184778053 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45435418 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180617763348992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚4987.076935671 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚327 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108766461 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170831590642176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7947992396800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944ÇËkÿ$2å- _wandbruntime‚4987  _runtime‚4987qºÍ\æÖ 0loss/SRqqt3/detection_loss‚0.839897244228973 /loss/FRt4/detection_loss‚3.0557602712741265 1loss/SRqqt2/detection_loss‚2.4236009969161105 0loss/SRqqt4/detection_loss‚3.242232084274292 /loss/FBt3/detection_loss‚0.7034347175698107 &loss/total_loss‚44.344067970602154 trainer/global_step‚3999 _runtime‚4987.078710216 0loss/FRt3/assignment_loss‚1.3304609153419733 0loss/FRt4/assignment_loss‚3.6164757242569556 2loss/SRqqt4/assignment_loss‚2.3061836050315336 2loss/FBt3/assignment_loss‚0.022810533266843642 1loss/SRqqt1/detection_loss‚1.9695816584255383 -loss/FBt1/detection_loss‚2.51087762759282 .loss/FBt2/detection_loss‚2.648998700082302 /loss/FRt2/detection_loss‚2.6227332055568695 /loss/FRt3/detection_loss‚1.6638639628654346 /loss/FBt4/detection_loss‚1.5986543624446943 2loss/SRqqt3/assignment_loss‚0.9966309515608324 /loss/FRt1/detection_loss‚2.2126940369606025 epoch‚3 % _timestamp‚1.7885340029251587e+09 0loss/FRt1/assignment_loss‚3.5527566141552396 2loss/SRqqt1/assignment_loss‚1.3463730228983837 1loss/SRqqt2/assignment_loss‚1.888646505199946 1loss/FBt1/assignment_loss‚0.15806471846043188 0loss/FRt2/assignment_loss‚3.1955506147876864 /loss/FBt2/assignment_loss‚0.147794074066951 0loss/FBt4/assignment_loss‚0.2899918233837997 _step‚328ôWA–açÛ 2loss/SRqqt3/assignment_loss‚0.9966309515608324 /loss/FRt1/detection_loss‚2.2126940369606025 epoch‚3 % _timestamp‚1.7885340029251587e+09 0loss/FRt1/assignment_loss‚3.5527566141552396 2loss/SRqqt1/assignment_loss‚1.3463730228983837 1loss/SRqqt2/assignment_loss‚1.888646505199946 1loss/FBt1/assignment_loss‚0.15806471846043188 0loss/FRt2/assignment_loss‚3.1955506147876864 /loss/FBt2/assignment_loss‚0.147794074066951 0loss/FBt4/assignment_loss‚0.2899918233837997 _step‚328 0loss/SRqqt3/detection_loss‚0.839897244228973 /loss/FRt4/detection_loss‚3.0557602712741265 1loss/SRqqt2/detection_loss‚2.4236009969161105 0loss/SRqqt4/detection_loss‚3.242232084274292 /loss/FBt3/detection_loss‚0.7034347175698107 &loss/total_loss‚44.344067970602154 trainer/global_step‚3999 _runtime‚4987.078710216 0loss/FRt3/assignment_loss‚1.3304609153419733 0loss/FRt4/assignment_loss‚3.6164757242569556 2loss/SRqqt4/assignment_loss‚2.3061836050315336 2loss/FBt3/assignment_loss‚0.022810533266843642 1loss/SRqqt1/detection_loss‚1.9695816584255383 -loss/FBt1/detection_loss‚2.51087762759282 .loss/FBt2/detection_loss‚2.648998700082302 /loss/FRt2/detection_loss‚2.6227332055568695 /loss/FRt3/detection_loss‚1.6638639628654346 /loss/FBt4/detection_loss‚1.5986543624446943È‹+**è‚(:„ ú¹ëÔúδ– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.57' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.714 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ q|é‚(:ô ú¹ëԠʲ™) disk./.usagePercent‚79.06539529160773 disk.sda.in‚0" disk.nvme0n1.in‚20083.66015625' disk.nvme0n1.out‚106324.94287109375$ disk.nvme0c0n1.in‚20083.66015625 disk.drbd1260.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 158.796875 disk.sda3.in‚0$ disk.dm-0.out‚106324.94287109375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.7734375$ memory_percent‚94.65470314025879' proc.memory.availableMB‚ 875.7734375) disk.nvme0c0n1.out‚106324.94287109375 network.recv‚ 1.660433e+06 disk.sda2.out‚0 disk.sda3.out‚ 158.796875) proc.memory.percent‚18.53957176208496 cpu‚25.32021542814781 network.sent‚ 9.604943e+06% disk./.usageGB‚1178.1315269470215 disk.sda2.in‚0 disk.dm-0.in‚20083.66015625 disk.dm-1.out‚ 158.796875 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3037.5234375%-ãÃê‚(:… ‰ºëÔ·…Ä— gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚100.371' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0742 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GL*}ë‚(:ò ‰ºëԲŊ˜ network.recv‚ 1.66663e+06$ disk./.usageGB‚1178.131534576416 disk.sda3.out‚ 159.6171875 disk.sda1.out‚0$ memory_percent‚94.65327262878418" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496( disk./.usagePercent‚79.0653958036228 disk.sda.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚20083.66015625 disk.sda3.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 876.0078125 cpu‚25.319475720643506& disk.nvme0n1.out‚106596.4345703125# disk.dm-0.out‚106596.4345703125 disk.dm-1.in‚0 disk.dm-1.out‚ 159.6171875 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚20083.66015625( disk.nvme0c0n1.out‚106596.4345703125 network.sent‚ 9.71553e+06 disk.sda.out‚ 159.6171875 disk.sda2.out‚0 disk.dm-0.in‚20083.66015625 disk.drbd1260.out‚ 0.77734375 proc.cpu.threads‚17S'—Šì‚(:„ ˜ºëÔ¡´– gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚127.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.416 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^’\étí‚(:é ˜ºëÔŸÄèš disk.sda2.out‚0' proc.memory.availableMB‚ 875.7578125 disk.dm-0.in‚20276.14453125 disk.sda.out‚160.25 disk.sda2.in‚0 cpu‚25.318895718326807 proc.cpu.threads‚17 network.sent‚ 9.718666e+06 network.recv‚ 1.669868e+06 disk.drbd1260.in‚0" disk.nvme0n1.in‚20276.14453125 disk.sda3.in‚0 disk.sda3.out‚160.25$ memory_percent‚94.65479850769043) proc.memory.percent‚18.53957176208496) disk./.usagePercent‚79.06384900610092) disk.nvme0c0n1.out‚107091.11474609375 disk.dm-1.out‚160.25 disk.drbd1260.out‚ 0.77734375 disk.sda.in‚0' disk.nvme0n1.out‚107091.11474609375" proc.memory.rssMB‚ 3037.5234375$ disk./.usageGB‚1178.108486175537$ disk.nvme0c0n1.in‚20276.14453125$ disk.dm-0.out‚107091.11474609375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0àn¬\‘î‚(:† §ºëÔœüߘ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0358 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÏý„ï‚(:ù §ºëÔ›˜‚› disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 160.74609375( proc.memory.availableMB‚ 875.75390625 disk.drbd1260.out‚0.78125 network.sent‚ 9.724779e+06) disk./.usagePercent‚79.06541551620295% disk./.usageGB‚1178.1318283081055) disk.nvme0c0n1.out‚107350.76416015625 disk.dm-0.in‚20276.14453125$ disk.dm-0.out‚107350.76416015625" disk.nvme0n1.in‚20276.14453125 disk.sda1.out‚0 disk.sda2.in‚0' disk.nvme0n1.out‚107350.76416015625 disk.sda3.out‚ 160.74609375) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17 disk.sda1.in‚0 network.recv‚ 1.674645e+06 disk.dm-1.out‚ 160.74609375 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚94.65482234954834" proc.memory.rssMB‚ 3037.5234375 cpu‚25.31834529610783$ disk.nvme0c0n1.in‚20276.14453125 disk.dm-1.in‚0µà^2ð- _wandbruntime‚5045  _runtime‚5045YÆñÀ _runtime‚5045.210731214 _step‚329 & lr-AdamW/pg1‚0.0006868685796571226 & lr-AdamW/pg2‚0.0006868685796571226 trainer/global_step‚4049 $ _timestamp‚1.788534061057024e+09ÅKËòÅ trainer/global_step‚4049 $ _timestamp‚1.788534061057024e+09 _runtime‚5045.210731214 _step‚329 & lr-AdamW/pg1‚0.0006868685796571226 & lr-AdamW/pg2‚0.0006868685796571226ÉÁøy2ó- _wandbruntime‚5045  _runtime‚5045Å^cß'ôÙO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8044010395648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164808614057984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 % _timestamp‚1.7885340610586064e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164808596055040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110077695 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29231559 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156842310289488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76933437 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8182468087808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186984788 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8182183759872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172852606450688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45964716 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76933437 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8044010395648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75195599 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8021209293636 trainer/global_step‚4049 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172852606450688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174575573526016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164808596055040 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172852858122240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29230883 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156842327328848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8021438274720 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110077695 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174575549585920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76933439 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172852858122240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182757733345792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164808614057984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45964718¢!¨õÞO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 % _timestamp‚1.7885340610586064e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164808596055040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110077695 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29231559 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156842310289488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76933437 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.smallÚ <#_pool.allocated‚ 8182468087808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186984788 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8182183759872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172852606450688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45964716 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76933437 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8044010395648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75195599 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8021209293636 trainer/global_step‚4049 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172852606450688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174575573526016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164808596055040 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172852858122240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29230883 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156842327328848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8021438274720 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110077695 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174575549585920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76933439 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172852858122240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182757733345792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164808614057984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45964718 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8044010395648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164808614057984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532Ê-}rý2ö- _wandbruntime‚5046  _runtime‚5046~»õ³ê&÷äM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8045974421504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚182801726751744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164903008429534 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164847882092032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164903254450186 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156879857686272 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156879840646912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29238436 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110078043 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187056127 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187029755 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110078043 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76951714 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110104413 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29237788 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚331 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172894108198400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187056127 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172894108198400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76951714 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75213956 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8184469255680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187029755 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164847882092032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76951712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45975518 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45975520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885340625737653e+09 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75213306 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8023167782622 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8046208103424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8046208103424 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110104413 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174617233555968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B0ï&øéM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187029755 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110078043 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76951714 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110104413 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29237788 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚331 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172894108198400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187056127 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172894108198400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76951714 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75213956 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8184469255680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187029755 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164847882092032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76951712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45975518 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45975520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885340625737653e+09 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75213306 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8023167782622 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8046208103424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8046208103424 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110104413 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174617233555968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8045974421504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚182801726751744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164903008429534 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164847882092032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164903254450186 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156879857686272 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156879840646912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29238436 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110078043 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187056127 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ËÈÑÿF2ù- _wandbruntime‚5046  _runtime‚50465~+ÓYúÓ /loss/SRqqt4/detection_loss‚2.50278183392116 /loss/FBt4/detection_loss‚1.3325024809510935 _runtime‚5046.730527375 /loss/FRt1/detection_loss‚2.2874048674428784 1loss/SRqqt1/detection_loss‚1.7503707643519058 %loss/total_loss‚45.45355351502708 trainer/global_step‚4049 0loss/FRt3/assignment_loss‚1.5251405003630085 2loss/SRqqt4/assignment_loss‚2.5152905074258647 0loss/FBt1/assignment_loss‚0.3032126263511724 1loss/FBt2/assignment_loss‚0.18511336620705757 /loss/FRt1/assignment_loss‚3.566025611516592 1loss/SRqqt2/detection_loss‚2.5632418627324314 1loss/SRqqt2/assignment_loss‚1.825815890389292 /loss/FRt4/detection_loss‚3.3154752097631754 /loss/FBt1/detection_loss‚2.4015909406271847 epoch‚3 /loss/FRt4/assignment_loss‚4.219420978897496 2loss/SRqqt1/assignment_loss‚1.4470170745824247 .loss/FBt2/detection_loss‚2.618006553914812 /loss/FBt3/detection_loss‚0.7897684389193144 _step‚332 2loss/FBt3/assignment_loss‚0.000276272516916671 .loss/FRt2/detection_loss‚2.889517649014791 0loss/FRt2/assignment_loss‚3.3190609792868297 0loss/FBt4/assignment_loss‚0.2973956533079174 .loss/FRt3/detection_loss‚1.672356842039728 0loss/SRqqt3/detection_loss‚0.974983386133077 % _timestamp‚1.7885340625770195e+09 2loss/SRqqt3/assignment_loss‚1.1517832243709631?ÿ^ûØ /loss/FRt4/assignment_loss‚4.219420978897496 2loss/SRqqt1/assignment_loss‚1.4470170745824247 .loss/FBt2/detection_loss‚2.618006553914812 /loss/FBt3/detection_loss‚0.7897684389193144 _step‚332 2loss/FBt3/assignment_loss‚0.000276272516916671 .loss/FRt2/detection_loss‚2.889517649014791 0loss/FRt2/assignment_loss‚3.3190609792868297 0loss/FBt4/assignment_loss‚0.2973956533079174 .loss/FRt3/detection_loss‚1.672356842039728 0loss/SRqqt3/detection_loss‚0.974983386133077 % _timestamp‚1.7885340625770195e+09 2loss/SRqqt3/assignment_loss‚1.1517832243709631 /loss/SRqqt4/detection_loss‚2.50278183392116 /loss/FBt4/detection_loss‚1.3325024809510935 _runtime‚5046.730527375 /loss/FRt1/detection_loss‚2.2874048674428784 1loss/SRqqt1/detection_loss‚1.7503707643519058 %loss/total_loss‚45.45355351502708 trainer/global_step‚4049 0loss/FRt3/assignment_loss‚1.5251405003630085 2loss/SRqqt4/assignment_loss‚2.5152905074258647 0loss/FBt1/assignment_loss‚0.3032126263511724 1loss/FBt2/assignment_loss‚0.18511336620705757 /loss/FRt1/assignment_loss‚3.566025611516592 1loss/SRqqt2/detection_loss‚2.5632418627324314 1loss/SRqqt2/assignment_loss‚1.825815890389292 /loss/FRt4/detection_loss‚3.3154752097631754 /loss/FBt1/detection_loss‚2.4015909406271847 epoch‚3ÌZÐnl‘ü‚(:† ¶ºëÔÇßš— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.077' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4154 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê;O>ý‚(:õ ¶ºëÔ¡ˆè˜ disk.drbd1260.out‚0.78125 disk.sda3.out‚ 161.1328125( proc.memory.availableMB‚ 875.71484375) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17 network.sent‚ 9.742684e+06 network.recv‚ 1.677859e+06% disk./.usageGB‚1178.1320571899414$ disk.nvme0c0n1.in‚20276.14453125 disk.dm-1.in‚0 disk.dm-1.out‚ 161.1328125 disk.sda1.in‚0 disk.sda.in‚0' disk./.usagePercent‚79.065430876655 disk.dm-0.in‚20276.14453125 disk.sdaÕ€.out‚ 161.1328125 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚20276.14453125' disk.nvme0n1.out‚107605.81494140625 disk.sda3.in‚0) disk.nvme0c0n1.out‚107605.81494140625 disk.sda1.out‚0$ memory_percent‚94.65506076812744" proc.memory.rssMB‚ 3037.5234375 cpu‚25.317724457268188$ disk.dm-0.out‚107605.81494140625 disk.drbd1260.in‚0u¿1dœþ‚(:‘ źëÔÎÀø• gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.351' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.470200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥7-¿†ÿ‚(:û źëÔôš' disk./.usagePercent‚79.065430876655 disk.dm-0.in‚20276.14453125 disk.drbd1260.out‚ 0.78515625 disk.sda.out‚ 161.48828125% disk./.usageGB‚1178.1320571899414 disk.sda3.out‚ 161.48828125$ disk.dm-0.out‚107907.90478515625 disk.dm-1.out‚ 161.48828125 disk.sda1.in‚0 disk.sda2.in‚0' disk.nvme0n1.out‚107907.90478515625$ memory_percent‚94.65672969818115$ disk.nvme0c0n1.in‚20276.14453125 disk.drbd1260.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 875.44140625" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 cpu‚25.317067910312964 disk.sda3.in‚0) disk.nvme0c0n1.out‚107907.90478515625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚20276.14453125 proc.cpu.threads‚17 network.sent‚ 9.838413e+06 network.recv‚ 1.684802e+06ªy=M€‚(:… ÔºëԸפ– gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.453 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…‡Ÿƒ‚(:ø ÔºëÔ±ð™) disk./.usagePercent‚79.03767761188307$ disk.nvme0c0n1.in‚20440.62890625 disk.dm-0.in‚20440.62890625 disk.sda1.in‚0 disk.drbd1260.out‚ 0.78515625 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 9.84345e+06 disk.dm-1.in‚0 disk.sda.in‚0$ memory_percent‚94.65525150299072 cpu‚25.31653587150214 network.recv‚ 1.688215e+06) disk.nvme0c0n1.out‚107908.33837890625$ disk.dm-0.out‚107908.33837890625 disk.dm-1.out‚ 161.8984375 disk.sda1.out‚0 disk.sda.out‚ 161.8984375" disk.nvme0n1.in‚20440.62890625( proc.memory.availableMB‚ 875.68359375% disk./.usageGB‚1177.7185134887695 disk.drbd1260.in‚0' disk.nvme0n1.out‚107908.33837890625 disk.sda3.in‚0 disk.sda3.out‚ 161.8984375" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17žâ¾©‘‚‚(:† ãºëÔŒ¥˜ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û&Œzƒ‚(:ö ãºëÔÁ¦‡¨) disk./.usagePercent‚79.10504676254101( disk.nvme0c0n1.out‚108350.8916015625 disk.drbd1260.out‚ 0.7890625 disk.sda.out‚ 162.2734375 disk.sda2.out‚0 cpu‚25.315954910412195% disk./.usageGB‚1178.7223625183105# disk.dm-0.out‚108350.8916015625 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚108350.8916015625 proc.cpu.threads‚17 network.sent‚ 9.849633e+06 network.recv‚ 1.692918e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 162.2734375 disk.sda1.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 875.17578125" proc.memory.rssMB‚ 3037.5234375$ disk.nvme0c0n1.in‚20440.62890625 disk.dm-0.in‚20440.62890625 disk.sda.in‚0" disk.nvme0n1.in‚20440.62890625 disk.sda3.in‚0 disk.sda3.out‚ 162.2734375$ memory_percent‚94.65835094451904) proc.memory.percent‚18.53957176208496`<2„- _wandbruntime‚5104  _runtime‚5104ull®Æ…À _step‚333 & lr-AdamW/pg1‚0.0006825301781922899 & lr-AdamW/pg2‚0.0006825301781922899 trainer/global_step‚4099 $ _timestamp‚1.788534120503868e+09 _runtime‚5104.657691184VæºóˆŠ& lr-AdamW/pg1‚0.0006825301781922899 & lr-AdamW/pg2‚0.0006825301781922899 trainer/global_step‚4099 $ _timestamp‚1.788534120503868e+09 _runtime‚5104.657691184 _step‚333Íûùv2‡- _wandbruntime‚5104  _runtime‚5104·ã'à'ˆÚO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176659748088320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8282355386880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111389303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8142459770880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166772915907584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166837975494832 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189236490 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77847189 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176659772028416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 trainer/global_step‚4099 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166772897904640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76082757 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77847187 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111415647 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174915124006912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29577939 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174915375678464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174915124006912 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8142226102272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76082071 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158718845200048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8119147334144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184942127415296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8142459770880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _runtime‚5104.659422036 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77847187 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176659772028416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 trainer/global_step‚4099 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166772897904640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76082757 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77847187 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111415647 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174915124006912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29577939 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174915375678464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174915124006912 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8142226102272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76082071 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158718845200048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8119147334144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184942127415296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8142459770880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _runtime‚5104.659422036 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77847187 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ÎÙœu,2Š- _wandbruntime‚5106  _runtime‚5106Ê&ä‹äM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8144190128128 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8144423810048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29585078 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158756358518112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77865462 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111415995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8121334804422 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166812183941632 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189307829 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46515620 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8284338137600 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174956625754624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189307829 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885341222983177e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76100035 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176701455998464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176701432058368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111442365 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189281457 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8144190128128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocationszYK .current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174956374069760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184985794136064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184985485881344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184985794136064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184985485881344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189307829 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46515620 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8284338137600 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174956625754624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189307829 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885341222983177e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76100035 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176701455998464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176701432058368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111442365 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189281457 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8144190128128Ï"]62- _wandbruntime‚5106  _runtime‚5106Fèvå\ŽÖ epoch‚3 % _timestamp‚1.7885341223022738e+09 _runtime‚5106.455763333 0loss/FBt2/assignment_loss‚0.0754940679089297 .loss/FRt4/detection_loss‚3.093093898296356 /loss/FBt2/detection_loss‚2.2355772902568183 &loss/total_loss‚43.480876308502076 0loss/FRt2/assignment_loss‚3.9767459210227516 /loss/FRt3/assignment_loss‚1.654282037789623 0loss/FBt1/assignment_loss‚0.4610514806116482 /loss/FBt4/detection_loss‚1.7130555198527873 0loss/FRt1/assignment_loss‚3.6697714683555422 1loss/SRqqt2/assignment_loss‚1.648355137705803 /loss/FRt3/detection_loss‚1.4101754845275234 /loss/FBt3/detection_loss‚0.9815468869075693 2loss/SRqqt3/assignment_loss‚1.2126489250126098 2loss/SRqqt4/assignment_loss‚2.0431692567653954 1loss/SRqqt3/detection_loss‚0.9177474408248149 _step‚336 3loss/FBt3/assignment_loss‚4.397683337340015e-05 0loss/FBt4/assignment_loss‚0.2510047315942927 .loss/FRt1/detection_loss‚2.159923833040966 2loss/SRqqt1/assignment_loss‚1.2851356241991743 1loss/SRqqt2/detection_loss‚2.3980353260040284 /loss/FBt1/detection_loss‚1.8450509714228767 trainer/global_step‚4099 0loss/FRt4/assignment_loss‚3.0380920934677125 .loss/FRt2/detection_loss‚2.795766755061991 0loss/SRqqt1/detection_loss‚1.820964553952217 0loss/SRqqt4/detection_loss‚2.794143627087275JüxaÛ /loss/FBt1/detection_loss‚1.8450509714228767 trainer/global_step‚4099 0loss/FRt4/assignment_loss‚3.0380920934677125 .loss/FRt2/detection_loss‚2.795766755061991 0loss/SRqqt1/detection_loss‚1.820964553952217 0loss/SRqqt4/detection_loss‚2.794143627087275 epoch‚3 % _timestamp‚1.7885341223022738e+09 _runtime‚5106.455763333 0loss/FBt2/assignment_loss‚0.0754940679089297 .loss/FRt4/detection_loss‚3.093093898296356 /loss/FBt2/detection_loss‚2.2355772902568183 &loss/total_loss‚43.480876308502076 0loss/FRt2/assignment_loss‚3.9767459210227516 /loss/FRt3/assignment_loss‚1.654282037789623 0loss/FBt1/assignment_loss‚0.4610514806116482 /loss/FBt4/detection_loss‚1.7130555198527873 0loss/FRt1/assignment_loss‚3.6697714683555422 1loss/SRqqt2/assignment_loss‚1.648355137705803 /loss/FRt3/detection_loss‚1.4101754845275234 /loss/FBt3/detection_loss‚0.9815468869075693 2loss/SRqqt3/assignment_loss‚1.2126489250126098 2loss/SRqqt4/assignment_loss‚2.0431692567653954 1loss/SRqqt3/detection_loss‚0.9177474408248149 _step‚336 3loss/FBt3/assignment_loss‚4.397683337340015e-05 0loss/FBt4/assignment_loss‚0.2510047315942927 .loss/FRt1/detection_loss‚2.159923833040966 2loss/SRqqt1/assignment_loss‚1.2851356241991743 1loss/SRqqt2/detection_loss‚2.3980353260040284ÐŒ9…œ‚(:‘ òºëÔ— gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚124.912' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.982400000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q‹c°w‘‚(:ì òºëÔíÔ˜ disk.dm-1.in‚0 disk.sda1.out‚0 cpu‚25.315389814396422 network.recv‚ 1.695942e+06 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚108754.5107421875 disk.dm-0.in‚20440.62890625 disk.sda1.in‚0$ memory_percent‚94.65992450714111" proc.memory.rssMB‚ 3037.5234375% disk./.usageGB‚1178.7188186645508 disk.sda.out‚162.6875" disk.nvme0n1.in‚20440.62890625& disk.nvme0n1.out‚108754.5107421875$ disk.nvme0c0n1.in‚20440.62890625 disk.dm-1.out‚162.6875( proc.memory.availableMB‚ 874.91796875 proc.cpu.threads‚17 network.sent‚ 9.854472e+06( disk./.usagePercent‚79.1048089315417 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7890625 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚162.6875) proc.memory.percent‚18.53957176208496# disk.dm-0.out‚108754.5107421875£·cœ’‚(:‘ »ëÔ´¡Ä˜ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.706599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î+M1“‚(:ö »ëÔó≜$ disk.nvme0c0n1.in‚20440.62890625# disk.dm-0.out‚108985.9033203125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda.out‚ 163.17578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 9.9634e+06 network.recv‚ 1.702918e+06$ disk./.usageGB‚1178.742286682129( disk.nvme0c0n1.out‚108985.9033203125 disk.sda1.out‚0 disk.sda.in‚0" disk.nvme0n1.in‚20440.62890625 disk.sda3.out‚ 163.17578125) disk./.usagePercent‚79.10638388989236 disk.dm-0.in‚20440.62890625& disk.nvme0n1.out‚108985.9033203125$ memory_percent‚94.65851783752441' proc.memory.availableMB‚ 875.1484375" proc.memory.rssMB‚ 3037.5234375 cpu‚25.314772774949347 disk.dm-1.out‚ 163.17578125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.79296875) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17n£èl”‚(:á »ëÔ”ò°˜ disk.nvme0n1.in‚ 20443.65625& disk.nvme0n1.out‚109345.2978515625 disk.dm-1.in‚0 disk.dm-1.out‚163.8125 disk.sda.out‚163.8125 disk.sda3.out‚163.8125! disk.nvme0c0n1.in‚ 20443.65625 disk.dm-0.in‚ 20443.65625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.79296875& proc.memory.availableMB‚ 875.640625 cpu‚25.31428464566527) disk./.usagePercent‚79.10638388989236 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 9.968238e+06 disk.sda2.out‚0# disk.dm-0.out‚109345.2978515625$ memory_percent‚94.65551376342773" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496( disk.nvme0c0n1.out‚109345.2978515625 network.recv‚ 1.706457e+06$ disk./.usageGB‚1178.742286682129 disk.sda2.in‚0^ÍZ‘•‚(:† »ëÔïÃè™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.268' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6536 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð>¯'‘–‚(:† Ÿ»ëÔ¯¥½˜ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g±”—‚(:ô Ÿ»ëÔÃÍÔœ disk.dm-0.in‚20578.53515625 disk.dm-1.in‚0( proc.memory.availableMB‚ 874.65234375) proc.memory.percent‚18.53957176208496$ disk.nvme0c0n1.in‚20578.53515625) disk.nvme0c0n1.out‚109408.20361328125$ disk.dm-0.out‚109408.20361328125 disk.sda1.in‚0 disk.sda.out‚ 164.359375 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 3037.5234375 network.sent‚ 9.974414e+06 network.recv‚ 1.711051e+06$ disk./.usageGB‚1178.742286682129 disk.sda1.out‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚20578.53515625' disk.nvme0n1.out‚109408.20361328125 disk.sda3.in‚0 disk.dm-1.out‚ 164.359375 disk.drbd1260.out‚ 0.79296875 disk.sda.in‚0 disk.sda3.out‚ 164.359375" memory_percent‚94.661545753479 cpu‚25.313623427898087 proc.cpu.threads‚17) disk./.usagePercent‚79.10638388989236Õì˜2˜- _wandbruntime‚5162  _runtime‚5162ð“‡AÅ™¿ & lr-AdamW/pg1‚0.0006781346287482269 & lr-AdamW/pg2‚0.0006781346287482269 trainer/global_step‚4149 # _timestamp‚1.78853417874248e+09 _runtime‚5162.896116872 _step‚337íÊCÊšÄ & lr-AdamW/pg1‚0.0006781346287482269 & lr-AdamW/pg2‚0.0006781346287482269 trainer/global_step‚4149 # _timestamp‚1.78853417874248e+09 _runtime‚5162.896116872 _step‚337Ñ·âuÒ2›- _wandbruntime‚5162  _runtime‚5162ž¦à'œÚO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168737217757184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78760937 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8382139159552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191488192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168737217757184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112753599 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8217085374652 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191514538 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112753599 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381854831616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7885341787438047e+09 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178743946590720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8217314355736 _runtime‚5162.897627049 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29928833 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112727255 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8240441808896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176977893234688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178743970530816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168812431406540 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112727255 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168812677426984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78760939 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚168737199754240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47044918 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168737199754240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187125801422336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160595363071248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176977641563136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8240441808896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176977641563136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78760939 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8240675477504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160595346031888 trainer/global_step‚4149 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176977893234688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8240675477504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76973075 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112727255 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8240441808896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176977893234688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178743970530816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168812431406540 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112727255 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168812677426984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78760939 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚168737199754240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47044918 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168737199754240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187125801422336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160595363071248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176977641563136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8240441808896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176977641563136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78760939 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8240675477504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160595346031888 trainer/global_step‚4149 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176977893234688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8240675477504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76973075 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112780317 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178785654500864 _runtime‚5164.061643272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160632876389312 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112753947 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8219043863638 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47055720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191533159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76991075 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160632893428672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8242405834752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168776503794176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8384139848704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29936037 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187169483997696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78779214 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _step‚339 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177018891625984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8242639516672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78779214 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78779212 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76991757 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8242639516672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191559531 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168851920252950 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191559531 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177019143310848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177019143310848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8242405834752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78779212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168852166273602 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191533159 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8383853436928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29935357 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885341799070742e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112753947 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47055718 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76991075 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160632893428672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8242405834752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168776503794176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8384139848704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29936037 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187169483997696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78779214 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _step‚339 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177018891625984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8242639516672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78779214 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78779212 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76991757 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8242639516672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191559531 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168851920252950 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191559531 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177019143310848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177019143310848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8242405834752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78779212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168852166273602 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191533159 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8383853436928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29935357 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885341799070742e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112753947 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47055718 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112780317 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178785654500864 _runtime‚5164.061643272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160632876389312 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112753947 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8219043863638 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47055720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191533159Ó袩2¡- _wandbruntime‚5164  _runtime‚5164‡*Z¢Ô .loss/SRqqt4/detection_loss‚2.2060978177822 .loss/FRt1/detection_loss‚2.294028455018997 epoch‚3 2loss/SRqqt1/assignment_loss‚1.6405017878084134 /loss/FRt2/assignment_loss‚3.808093100786209 2loss/SRqqt4/assignment_loss‚1.9925284069595914 0loss/SRqqt1/detection_loss‚1.817027596756816 %loss/total_loss‚44.17134863241397 /loss/FBt3/detection_loss‚0.9592927990383223 2loss/SRqqt3/assignment_loss‚1.1248884602951317 0loss/FBt1/assignment_loss‚0.3791662013151533 2loss/FBt3/assignment_loss‚0.026993080054952514 1loss/SRqqt2/detection_loss‚2.5656504333019257 /loss/FRt4/assignment_loss‚2.895744725509926 /loss/FBt1/detection_loss‚2.3653523325920105 /loss/FRt1/assignment_loss‚4.178600239753723 1loss/FBt4/assignment_loss‚0.13901992596674972 /loss/FRt3/detection_loss‚1.5507793765313722 /loss/FBt4/detection_loss‚1.6865439299080107 trainer/global_step‚4149 % _timestamp‚1.7885341799101346e+09 2loss/SRqqt2/assignment_loss‚2.1097048200093784 .loss/FRt2/detection_loss‚3.132493080837386 /loss/FRt4/detection_loss‚2.8608864347139993 /loss/FBt2/detection_loss‚1.8935668062080036 _runtime‚5164.063502667 _step‚340 0loss/FRt3/assignment_loss‚1.5303558449487429 1loss/FBt2/assignment_loss‚0.09185075717815297 1loss/SRqqt3/detection_loss‚0.9221822191387988¢ªú_£Ù 0loss/SRqqt1/detection_loss‚1.817027596756816 %loss/total_loss‚44.17134863241397 /loss/FBt3/detection_loss‚0.9592927990383223 2loss/SRqqt3/assignment_loss‚1.1248884602951317 0loss/FBt1/assignment_loss‚0.3791662013151533 2loss/FBt3/assignment_loss‚0.026993080054952514 1loss/SRqqt2/detection_loss‚2.5656504333019257 /loss/FRt4/assignment_loss‚2.895744725509926 /loss/FBt1/detection_loss‚2.3653523325920105 /loss/FRt1/assignment_loss‚4.178600239753723 1loss/FBt4/assignment_loss‚0.13901992596674972 /loss/FRt3/detection_loss‚1.5507793765313722 /loss/FBt4/detection_loss‚1.6865439299080107 trainer/global_step‚4149 % _timestamp‚1.7885341799101346e+09 2loss/SRqqt2/assignment_loss‚2.1097048200093784 .loss/FRt2/detection_loss‚3.132493080837386 /loss/FRt4/detection_loss‚2.8608864347139993 /loss/FBt2/detection_loss‚1.8935668062080036 _runtime‚5164.063502667 _step‚340 0loss/FRt3/assignment_loss‚1.5303558449487429 1loss/FBt2/assignment_loss‚0.09185075717815297 1loss/SRqqt3/detection_loss‚0.9221822191387988 .loss/SRqqt4/detection_loss‚2.2060978177822 .loss/FRt1/detection_loss‚2.294028455018997 epoch‚3 2loss/SRqqt1/assignment_loss‚1.6405017878084134 /loss/FRt2/assignment_loss‚3.808093100786209 2loss/SRqqt4/assignment_loss‚1.9925284069595914ÔÙnת¤‚(:… ®»ëÔìÐ÷– gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íßT Ý¥‚(:ñ ®»ëÔþ×™ network.sent‚ 9.978417e+06 network.recv‚ 1.71366e+06 disk.dm-0.in‚20578.53515625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚20578.53515625$ disk.dm-0.out‚109826.10595703125 disk.dm-1.out‚ 164.765625 disk.drbd1260.out‚0.796875 disk.sda.out‚ 164.765625' disk.nvme0n1.out‚109826.10595703125$ memory_percent‚94.65723037719727 proc.cpu.threads‚17( disk./.usagePercent‚79.1048089315417) disk.nvme0c0n1.out‚109826.10595703125 disk.sda1.in‚0) proc.memory.percent‚18.53957176208496% disk./.usageGB‚1178.7188186645508 disk.sda.in‚0" disk.nvme0n1.in‚20578.535Ç矟15625 disk.sda3.in‚0 disk.sda3.out‚ 164.765625& proc.memory.availableMB‚ 875.359375" proc.memory.rssMB‚ 3037.5234375 cpu‚25.313133681100343ÆìÈN‘¦‚(:† ½»ëÔÝÔÿ• gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8492 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aµŠ§‚(:ô ½»ëÔåÖ¯š" disk.dm-0.out‚110167.099609375 disk.dm-1.out‚ 165.19921875 disk.sda.out‚ 165.19921875% disk.nvme0n1.out‚110167.099609375$ memory_percent‚94.65889930725098' disk.nvme0c0n1.out‚110167.099609375 disk.dm-0.in‚20578.53515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚20578.53515625) proc.memory.percent‚18.53957176208496 network.recv‚ 1.720394e+06) disk./.usagePercent‚79.10638388989236$ disk.nvme0c0n1.in‚20578.53515625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda3.in‚0$ disk./.usageGB‚1178.742286682129 disk.drbd1260.out‚0.796875 disk.sda3.out‚ 165.19921875' proc.memory.availableMB‚ 875.0859375" proc.memory.rssMB‚ 3037.5234375 cpu‚25.312515334158597 proc.cpu.threads‚17 network.sent‚ 1.0087243e+07è—&‘¨‚(:† Ì»ëÔþ²­– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0026 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é5鼩‚(:ô Ì»ëÔøØ‹› disk.sda3.in‚0 disk.sda1.in‚0 network.recv‚ 1.72389e+06) disk./.usagePercent‚79.04021003841179 disk.sda2.in‚0% disk.nvme0n1.out‚110488.716796875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.80078125& proc.memory.availableMB‚ 875.328125$ disk./.usageGB‚1177.756248474121 disk.sda2.out‚0 disk.sda3.out‚ 165.78515625$ disk.nvme0c0n1.in‚20578.53515625" disk.dm-0.out‚110488.716796875 disk.sda1.out‚0" proc.memory.rssMB‚ 3037.5234375 proc.cpu.threads‚17 disk.sda.in‚0" disk.nvme0n1.in‚20578.53515625' disk.nvme0c0n1.out‚110488.716796875 disk.dm-0.in‚20578.53515625 disk.dm-1.out‚ 165.78515625$ memory_percent‚94.65742111206055) proc.memory.percent‚18.53957176208496 cpu‚25.312043016050172 network.sent‚ 1.0092112e+07 disk.sda.out‚ 165.78515625’ëôª‚(:… Û»ëÔ–ÔŸ– gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8316 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹•ËF…«‚(:ú Û»ëÔ©éÜ™ disk.sda2.in‚0 disk.sda3.out‚ 166.1484375) disk.nvme0c0n1.out‚110958.79833984375$ disk.dm-0.out‚110958.79833984375$ memory_percent‚94.65744495391846( proc.memory.availableMB‚ 875.32421875$ disk./.usageGB‚1177.756248474121" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 cpu‚25.311451463001234 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0' disk.nvme0n1.out‚110958.79833984375$ disk.nvme0c0n1.in‚20712.83984375 disk.dm-0.in‚20712.83984375 disk.dm-1.in‚0 disk.dm-1.out‚ 166.1484375 network.sent‚ 1.0100145e+07 network.recv‚ 1.728648e+06) disk./.usagePercent‚79.04021003841179 disk.drbd1260.out‚ 0.80078125 disk.sda.in‚0 disk.sda.out‚ 166.1484375" disk.nvme0n1.in‚20712.83984375 disk.sda3.in‚0 disk.drbd1260.in‚0ü˜eå2¬- _wandbruntime‚5223  _runtime‚5223Å5ÙUÇ­Á _runtime‚5223.122028142 _step‚341 & lr-AdamW/pg1‚0.0006736828473508499 & lr-AdamW/pg2‚0.0006736828473508499 trainer/global_step‚4199 % _timestamp‚1.7885342389681864e+09ƒÍ«Ì®Æ % _timestamp‚1.7885342389681864e+09 _runtime‚5223.122028142 _step‚341 & lr-AdamW/pg1‚0.0006736828473508499 & lr-AdamW/pg2‚0.0006736828473508499 trainer/global_step‚4199Õˆïü¯2¯- _wandbruntime‚5223  _runtime‚5223îl²&ß'°ÙO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114091551 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30277377 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179040410790912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170786887318248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8338657515520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189310137539072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180828169033216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 _step‚342 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8338657515520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179040159119360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8338891184128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189309829271040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8315023415160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8338891184128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8315252396244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79674689 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481684177920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193739894 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170701519606784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114065207 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47585016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481968505856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79674687 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚170787133338692 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170701519606784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79674687 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179040159119360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114091551 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170701501603840 trainer/global_step‚4199 $ _timestamp‚1.788534238969819e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚162471880942448LbÃ;ä'±ÞO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170701501603840 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179040410790912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180828145093120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114065207 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77862395 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77861721 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30276705 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30277377 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179040410790912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170786887318248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8338657515520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189310137539072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180828169033216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 _step‚342 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8338657515520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179040159119360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8338891184128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189309829271040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8315023415160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8338891184128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8315252396244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79674689 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481684177920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193739894 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170701519606784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114065207 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47585016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481968505856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79674687 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚170787133338692 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170701519606784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79674687 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179040159119360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114091551 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170701501603840 trainer/global_step‚4199 $ _timestamp‚1.788534238969819e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚162471880942448 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114091551 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193811233 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114091899 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8483970507776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179081409182208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193784861 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚162509394260512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77879217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47595818 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170740805643776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚343 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79692964 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8340855223296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114118269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180869829063168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193784861 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189353502673408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170740805643776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193811233 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179081660867072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189353823511040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8317210885438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179081660867072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30283399 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162509411299872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8340855223296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180869853003264 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114118269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114091899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170740787640832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170740787640832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.l¢–7 arge_pool.allocated‚47595820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8483673610240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚4199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8316981904146 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77879910 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚5224.465860423 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193811233 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114091899 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8483970507776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179081409182208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193784861 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚162509394260512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77879217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47595818 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170740805643776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚343 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79692964 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8340855223296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114118269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180869829063168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193784861 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189353502673408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170740805643776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193811233 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179081660867072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189353823511040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8317210885438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179081660867072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30283399 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162509411299872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8340855223296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180869853003264 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114118269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114091899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170740787640832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170740787640832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47595820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8483673610240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚4199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8316981904146 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77879910 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚5224.465860423 E$Là'ÄÚO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8436873222144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181102676675584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚172665803453440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8412961455668 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115403159 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78747766 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7885342983659446e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181102928347136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48125116 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80588437 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115403159 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4249 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196017942 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115429503 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191493945794048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78747093 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8437106890752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191494262450688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115429503 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182912343595520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80588439 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80588437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚172761343229956 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30621977 _runtime‚5282.519798355 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚172665821456384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182912367535616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181102928347136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚172665821456384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8581894915072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚164348381774288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172761589250400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚164348398813648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181102676675584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8437106890752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚172665821456384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182912367535616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181102928347136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚172665821456384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8581894915072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚164348381774288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172761589250400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚164348398813648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181102676675584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8437106890752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚172665803453440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8412961455668 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115403159 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78747766 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7885342983659446e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181102928347136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48125116 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80588437 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115403159 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4249 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736Ú¸»Ø 2Æ- _wandbruntime‚5284  _runtime‚528419Îeê&ÇäM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182954027565568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80606712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885343000660706e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196062935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80606714 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 trainer/global_step‚4249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172705089490432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8439070929920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191537623734784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191537936183808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182954051505664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚5284.220733219 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78765104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚172705089490432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80606712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172705107493376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8438837248000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196036563 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48135918 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8415148925946 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115456221 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172801078097018 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172800832076366 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196062935 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115456221 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8583596169216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚347 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196036563 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30629184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30628520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164385929171072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8438837248000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8583884678144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181144178423296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115429851 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78764438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181143926738432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80606714 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48135920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273&d‰ï&ÈéM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164385929171072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8438837248000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8583884678144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181144178423296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115429851 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78764438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181143926738432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80606714 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48135920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182954027565568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80606712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885343000660706e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196062935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80606714 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 trainer/global_step‚4249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172705089490432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8439070929920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191537623734784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191537936183808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182954051505664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚5284.220733219 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78765104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚172705089490432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80606712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172705107493376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8438837248000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196036563 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48135918 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8415148925946 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115456221 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172801078097018 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172800832076366 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196062935 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115456221 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8583596169216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚347 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196036563 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30629184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30628520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608Û]U›2É- _wandbruntime‚5284  _runtime‚5284W²*WÊÑ -loss/FRt1/detection_loss‚2.17891263625457 /loss/FRt3/detection_loss‚1.3676426628870624 0loss/SRqqt2/detection_loss‚2.655742738928114 trainer/global_step‚4249 2loss/FBt3/assignment_loss‚0.010338863393099551 1loss/FBt4/assignment_loss‚0.15712910376553607 /loss/FBt1/detection_loss‚2.5252616916384016 0loss/FRt1/assignment_loss‚3.4346152544021606 .loss/FRt2/detection_loss‚2.893593830090982 /loss/SRqqt4/detection_loss‚2.64251509308815 .loss/FBt2/detection_loss‚2.050435285819204 .loss/FBt3/detection_loss‚0.707035556808114 /loss/FBt2/assignment_loss‚0.422913451712786 0loss/SRqqt1/detection_loss‚1.900024088844657 /loss/FRt2/assignment_loss‚3.587617333288546 0loss/FRt3/assignment_loss‚1.6562133313535334 2loss/SRqqt1/assignment_loss‚1.2862299839034677 /loss/FBt4/detection_loss‚1.3920424950974328 %loss/total_loss‚42.97079480719299 $ _timestamp‚1.788534300069002e+09 2loss/SRqqt3/assignment_loss‚1.1223203992857687 epoch‚3 2loss/SRqqt4/assignment_loss‚1.9804243355989457 /loss/FRt4/detection_loss‚2.9897414264471633 _step‚348 0loss/FRt4/assignment_loss‚3.3251969166423962 2loss/SRqqt2/assignment_loss‚1.6351094501359122 1loss/SRqqt3/detection_loss‚0.8782809856787043 _runtime‚5284.222432782 1loss/FBt1/assignment_loss‚0.17145789212827975„{d\ËÖ 0loss/FRt1/assignment_loss‚3.4346152544021606 .loss/FRt2/detection_loss‚2.893593830090982 /loss/SRqqt4/detection_loss‚2.64251509308815 .loss/FBt2/detection_loss‚2.050435285819204 .loss/FBt3/detection_loss‚0.707035556808114 /loss/FBt2/assignment_loss‚0.422913451712786 0loss/SRqqt1/detection_loss‚1.900024088844657 /loss/FRt2/assignment_loss‚3.587617333288546 0loss/FRt3/assignment_loss‚1.6562133313535334 2loss/SRqqt1/assignment_loss‚1.2862299839034677 /loss/FBt4/detection_loss‚1.3920424950974328 %loss/total_loss‚42.97079480719299 $ _timestamp‚1.788534300069002e+09 2loss/SRqqt3/assignment_loss‚1.1223203992857687 epoch‚3 2loss/SRqqt4/assignment_loss‚1.9804243355989457 /loss/FRt4/detection_loss‚2.9897414264471633 _step‚348 0loss/FRt4/assignment_loss‚3.3251969166423962 2loss/SRqqt2/assignment_loss‚1.6351094501359122 1loss/SRqqt3/detection_loss‚0.8782809856787043 _runtime‚5284.222432782 1loss/FBt1/assignment_loss‚0.17145789212827975 -loss/FRt1/detection_loss‚2.17891263625457 /loss/FRt3/detection_loss‚1.3676426628870624 0loss/SRqqt2/detection_loss‚2.655742738928114 trainer/global_step‚4249 2loss/FBt3/assignment_loss‚0.010338863393099551 1loss/FBt4/assignment_loss‚0.15712910376553607 /loss/FBt1/detection_loss‚2.5252616916384016Ü?§œÌ‚(:‘ ¦¼ëÔÙЖ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.954' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.590799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X¨‹{Í‚(:𠦼ëÔ¡°­" proc.memory.rssMB‚ 3037.5234375) disk./.usagePercent‚79.04021746263027% disk./.usageGB‚1177.7563591003418& disk.nvme0c0n1.out‚112371.27734375 disk.dm-1.in‚0 network.sent‚ 1.0227813e+07! disk.dm-0.out‚112371.27734375 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 20885.2109375$ memory_percent‚94.65785026550293) proc.memory.percent‚18.53957176208496 cpu‚25.30876908776275 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 20885.2109375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.81640625 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚112371.27734375 disk.sda3.out‚ 168.86328125' proc.memory.availableMB‚ 875.2578125 network.recv‚ 1.750348e+06 disk.dm-0.in‚ 20885.2109375 disk.dm-1.out‚ 168.86328125 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 168.86328125 disk.sda3.in‚0#ÚµÓ7΂(:î µ¼ëÔäýƒ˜) proc.memory.percent‚18.5+oAËB3957176208496 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 875.5078125 cpu‚25.308282457414837 network.sent‚ 1.0336575e+07 network.recv‚ 1.757177e+06) disk./.usagePercent‚79.04021746263027# disk.nvme0n1.out‚112524.6015625 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 20885.2109375 disk.dm-0.in‚ 20885.2109375 disk.dm-0.out‚112524.6015625% disk./.usageGB‚1177.7563591003418 disk.sda.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 20885.2109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.81640625$ memory_percent‚94.65632438659668% disk.nvme0c0n1.out‚112524.6015625 disk.sda.out‚ 169.27734375 disk.sda3.out‚ 169.27734375 disk.dm-1.out‚ 169.27734375 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3037.5234375[Ò§šÏ‚(:… µ¼ëÔˆ•ü— gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.403' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8806 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eUL›Ð‚(:… ļëÔŠ‹€– gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.717 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ê6}Ñ‚(:ò ļëÔ¥™$ memory_percent‚94.65804100036621# disk.dm-0.out‚112793.8642578125 disk.dm-1.in‚0& disk.nvme0n1.out‚112793.8642578125' proc.memory.availableMB‚ 875.2265625" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 1.0343394e+07# disk.nvme0c0n1.in‚ 20885.2109375( disk.nvme0c0n1.out‚112793.8642578125 cpu‚25.307735546261387 network.recv‚ 1.76085e+06 disk.sda3.out‚ 169.6484375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 169.6484375 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚79.03864327230224% disk./.usageGB‚1177.7329025268555 disk.sda3.in‚0 disk.dm-0.in‚ 20885.2109375 disk.dm-1.out‚ 169.6484375 disk.drbd1260.out‚ 0.8203125! disk.nvme0n1.in‚ 20885.2109375a~„…‘Ò‚(:† Ó¼ëÔ—°ï• gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2244 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.’²wÓ‚(:ì Ó¼ëÔ‡•’› network.recv‚ 1.76518e+06% disk./.usageGB‚1177.7329025268555" disk.nvme0c0n1.in‚ 21036.109375( disk.nvme0c0n1.out‚113115.3798828125 disk.dm-0.in‚ 21036.109375 disk.dm-1.out‚ 170.109375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8203125) disk./.usagePercent‚79.03864327230224 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 21036.109375 disk.sda3.out‚ 170.109375$ memory_percent‚94.66109275817871 network.sent‚ 1.0347601e+07# disk.dm-0.out‚113115.3798828125 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 170.109375& disk.nvme0n1.out‚113115.3798828125' proc.memory.availableMB‚ 874.7265625) proc.memory.percent‚18.53957176208496 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 3037.5234375 cpu‚25.307228926290982 proc.cpu.threads‚17¤@Ò2Ô- _wandbruntime‚5343  _runtime‚5343qÏÅxÃÕ½ trainer/global_step‚4299 % _timestamp‚1.7885343589483376e+09 _runtime‚5343.102027128 _step‚349 $ lr-AdamW/pg1‚0.00066461431119975 $ lr-AdamW/pg2‚0.00066461431119975-¢ø;ÈÖ _runtime‚5343.102027128 _step‚349 $ lr-AdamW/pg1‚0.00066461431119975 $ lr-AdamW/pg2‚0.00066461431119975 trainer/global_step‚4299 % _timestamp‚1.7885343589483376e+09ÝÐ7KÅ2×- _wandbruntime‚5343  _runtime‚5343f<Æ»à'ØÚO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183165445903360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116767455 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8681716591104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30968406 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚5343.103428059 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174630105303040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81502189 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81502189 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚350 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174630123305984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116767455 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116741111 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193677961778176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116741111 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183165194231808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198269644 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885343589497447e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8535322597376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193678282629120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166224899645488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48665216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184996542097920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166224916684848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116767455 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8681716591104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30968406 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚5343.103428059 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174630105303040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81502189 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81502189 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚350 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174630123305984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116767455 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116741111 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193677961778176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116741111 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183165194231808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198269644 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885343589497447e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8535322597376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193678282629120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166224899645488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48665216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184996542097920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166224916684848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8683409889792 _step‚351 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8537052954624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79651027 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚5344.590114592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8537286636544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30975680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183206695979520 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116767803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174775534008726 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174775287988074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81520462 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116767803 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8537052954624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183206695979520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚4299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116794173 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81520462 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8512857985162 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48676018 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193721952601088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185038250008064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198314637 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166262447042272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185038226067968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174669391340032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.aˆŽ5]llocated‚81520464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885343604359334e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174669409342976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193721635957760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174669409342976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166262430002912 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8512857985162 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48676018 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193721952601088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185038250008064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198314637 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166262447042272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185038226067968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174669391340032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81520464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885343604359334e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174669409342976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193721635957760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174669409342976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166262430002912 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8683409889792 _step‚351 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8537052954624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79651027 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚5344.590114592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8537286636544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30975680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183206695979520 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116767803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174775534008726 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174775287988074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81520462 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104ß™1MÞ2Ý- _wandbruntime‚5344  _runtime‚5344ÔÄ¿Ó[ÞÕ % _timestamp‚1.7885343604384563e+09 ,loss/FRt4/detection_loss‚3.0009797723205 /loss/FBt1/detection_loss‚2.5809109807014465 .loss/FBt3/detection_loss‚1.043782813192317 _runtime‚5344.591812663 0loss/FRt1/assignment_loss‚3.9345809078734852 2loss/SRqqt3/assignment_loss‚1.0163737129868131 1loss/SRqqt4/detection_loss‚2.3463432769621573 .loss/FBt2/detection_loss‚2.123040430686053 /loss/FBt4/detection_loss‚1.7173143193639557 trainer/global_step‚4299 2loss/SRqqt4/assignment_loss‚1.7673470710794772 /loss/FRt1/detection_loss‚2.3379278481006627 1loss/SRqqt2/detection_loss‚2.8042364581064745 %loss/total_loss‚44.78301284767213 2loss/SRqqt2/assignment_loss‚1.7363033477555623 0loss/FBt1/assignment_loss‚0.2821553352717198 epoch‚3 _step‚352 0loss/FRt3/assignment_loss‚1.6381333515048027 3loss/FBt3/assignment_loss‚0.0006849189855742997 1loss/SRqqt1/detection_loss‚1.8764211528523023 /loss/FRt2/assignment_loss‚4.507725452383359 /loss/FRt4/assignment_loss‚2.672861196376659 2loss/SRqqt1/assignment_loss‚1.5203244971154735 1loss/FBt2/assignment_loss‚0.14407714501687544 1loss/SRqqt3/detection_loss‚0.9231394230616543 1loss/FBt4/assignment_loss‚0.06948900881343434 .loss/FRt2/detection_loss‚3.183777163426081 .loss/FRt3/detection_loss‚1.555083263735287H—`ßÚ /loss/FRt2/assignment_loss‚4.507725452383359 /loss/FRt4/assignment_loss‚2.672861196376659 2loss/SRqqt1/assignment_loss‚1.5203244971154735 1loss/FBt2/assignment_loss‚0.14407714501687544 1loss/SRqqt3/detection_loss‚0.9231394230616543 1loss/FBt4/assignment_loss‚0.06948900881343434 .loss/FRt2/detection_loss‚3.183777163426081 .loss/FRt3/detection_loss‚1.555083263735287 % _timestamp‚1.7885343604384563e+09 ,loss/FRt4/detection_loss‚3.0009797723205 /loss/FBt1/detection_loss‚2.5809109807014465 .loss/FBt3/detection_loss‚1.043782813192317 _runtime‚5344.591812663 0loss/FRt1/assignment_loss‚3.9345809078734852 2loss/SRqqt3/assignment_loss‚1.0163737129868131 1loss/SRqqt4/detection_loss‚2.3463432769621573 .loss/FBt2/detection_loss‚2.123040430686053 /loss/FBt4/detection_loss‚1.7173143193639557 trainer/global_step‚4299 2loss/SRqqt4/assignment_loss‚1.7673470710794772 /loss/FRt1/detection_loss‚2.3379278481006627 1loss/SRqqt2/detection_loss‚2.8042364581064745 %loss/total_loss‚44.78301284767213 2loss/SRqqt2/assignment_loss‚1.7363033477555623 0loss/FBt1/assignment_loss‚0.2821553352717198 epoch‚3 _step‚352 0loss/FRt3/assignment_loss‚1.6381333515048027 3loss/FBt3/assignment_loss‚0.0006849189855742997 1loss/SRqqt1/detection_loss‚1.8764211528523023àGg·3‘à‚(:† â¼ëÔÀÕȘ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5082 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DÉy‚á‚(:÷ â¼ëÔ–ž› disk.sda2.in‚0 disk.sda3.out‚ 170.65234375) proc.memory.percent‚18.53957176208496 network.sent‚ 1.0352475e+07 network.recv‚ 1.768389e+06) disk./.usagePercent‚79.03864352830976 disk.dm-1.in‚0' disk.nvme0n1.out‚113346.58935546875# memory_percent‚94.6582555770874( proc.memory.availableMB‚ 875.19140625" proc.memory.rssMB‚ 3037.5234375 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 21036.109375 disk.dm-0.in‚ 21036.109375$ disk.dm-0.out‚113346.58935546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 170.65234375 disk.sda2.out‚0 disk.nvme0n1.in‚ 21036.109375 disk.sda3.in‚0 cpu‚25.306695877213667 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.82421875% disk./.usageGB‚1177.7329063415527) disk.nvme0c0n1.out‚113346.58935546875 disk.dm-1.out‚ 170.65234375Íß­â‚(:„ ñ¼ëÔΓï” gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.455' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.691 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖsÄ®yã‚(:î ñ¼ëԣ줚 disk.sda2.in‚0 disk.sda3.out‚ 171.015625 network.sent‚ 1.0461389e+07 disk.dm-0.in‚ 21036.109375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0) disk.nvme0c0n1.out‚113816.41748046875$ proc.memory.availableMB‚874.9375" proc.memory.rssMB‚ 3037.5234375) disk./.usagePercent‚79.04021848666041% disk./.usageGB‚1177.7563743591309 disk.nvme0n1.in‚ 21036.109375' disk.nvme0n1.out‚113816.41748046875 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 21036.109375$ memory_percent‚94.65980529785156) proc.memory.percent‚18.53957176208496 cpu‚25.306180312079913 proc.cpu.threads‚17 network.recv‚ 1.775133e+06$ disk.dm-0.out‚113816.41748046875 disk.dm-1.in‚0 disk.dm-1.out‚ 171.015625 disk.drbd1260.out‚ 0.82421875 disk.sda.out‚ 171.015625^ŽÎ‘ä‚(:† €½ëÔÏÒ“– gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚124.381' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8762 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»Êä¯}å‚(:ò €½ëÔ껡™% disk./.usageGB‚1177.7674598693848 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 171.4296875" disk.nvme0c0n1.in‚ 21036.109375$ disk.dm-0.out‚113972.82177734375$ memory_percent‚94.65978145599365( proc.memory.availableMB‚ 874.94140625 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 21036.109375 cpu‚25.305665782321583 network.sent‚ 1.0466336e+07 disk.drbd1260.in‚0 disk.sda3.in‚0) disk.nvme0c0n1.out‚113972.82177734375 disk.dm-0.in‚ 21036.109375" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17 network.recv‚ 1.778246e+06( disk./.usagePercent‚79.0409624445549 disk.dm-1.in‚0 disk.dm-1.out‚ 171.4296875 disk.sda1.out‚0 disk.drbd1260.out‚0.828125 disk.sda.out‚ 171.4296875' disk.nvme0n1.out‚113972.82177734375ª† T‘æ‚(:† ½ëÔ”¦©— gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.603' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7206 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë*X}ç‚(:ò ½ëÔŒ­å› network.sent‚ 1.0472384e+07' disk.nvme0n1.out‚114274.53955078125 disk.sda3.out‚ 171.80859375% proc.memory.availableMB‚ 874.46875" proc.memory.rssMB‚ 3037.5234375 cpu‚25.30521176101146 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0% disk./.usageGB‚1177.7670288085938" disk.nvme0c0n1.in‚ 21091.203125$ disk.dm-0.out‚114274.53955078125 disk.sda3.in‚0$ memory_percent‚94.66266632080078 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 21091.203125) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17 disk.dm-1.out‚ 171.80859375 disk.drbd1260.out‚0.828125 network.recv‚ 1.782761e+06) disk./.usagePercent‚79.04093351570354) disk.nvme0c0n1.out‚114274.53955078125 disk.dm-0.in‚ 21091.203125 disk.dm-1.in‚0 disk.sda.out‚ 171.80859375‰t2è- _wandbruntime‚5403  _runtime‚5403GôjÊÆéÀ _step‚353 & lr-AdamW/pg1‚0.0006599994463153689 & lr-AdamW/pg2‚0.0006599994463153689 trainer/global_step‚4349 $ _timestamp‚1.788534419469036e+09 _runtime‚5403.622487815äÍeËêÅ $ _timestamp‚1.788534419469036e+09 _runtime‚5403.622487815 _step‚353 & lr-AdamW/pg1‚0.0006599994463153689 & lr-AdamW/pg2‚0.0006599994463153689 trainer/global_step‚4349áýgíP2ë- _wandbruntime‚5403  _runtime‚5403|$ñ#ß'ìÙO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176594407152640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195862031393792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8633304635392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8633304635392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31312126 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚354 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82415937 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187080740600320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8608837536684 $ _timestamp‚1.788534419470013e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80518128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200521346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185227963459584 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185227711788032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185227963459584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118079063 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚4349 _runtime‚5403.623714657 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118105407 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8781581412864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8781290793472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82415939 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185227711788032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176594425155584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49205316 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31312810 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82415937 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49205318 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8633538304000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8633538304000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176710501073816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176594407152640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118079063 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176710255053372 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8633304635392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8633304635392 UGDeviceStatsMonitor.on_train_baFê”i tch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31312126 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚354 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82415937 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187080740600320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8608837536684 $ _timestamp‚1.788534419470013e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80518128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200521346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185227963459584 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185227711788032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185227963459584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118079063 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚4349 _runtime‚5403.623714657 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118105407 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8781581412864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8781290793472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82415939 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185227711788032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176594425155584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49205316 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31312810 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82415937 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49205318 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8633538304000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8633538304000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176710501073816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176594407152640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118079063 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176710255053372 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200566339 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚5404.618304343 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185269213535744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8610796025670 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8611025006962 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200566339 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82434212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885344204640696e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176749989920434 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185269213535744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80535578 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80534892 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195906018637824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187122448510464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8783570127360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176749743899782 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200539967 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31318774 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8635268661248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118132125 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚355 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8635502343168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200539967 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187122424570368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176633711192576 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176633693189632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 trainer/global_step‚4349ï^¤ï&ðéM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195905712480256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8783287909888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168138947874112 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176633693189632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 trainer/global_step‚4349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82434214 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200566339 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚5404.618304343 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185269213535744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8610796025670 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8611025006962 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200566339 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82434212 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885344204640696e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176749989920434 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185269213535744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80535578 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80534892 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195906018637824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187122448510464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8783570127360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176749743899782 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200539967 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31318774 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8635268661248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118132125 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚355 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8635502343168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200539967 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187122424570368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176633711192576 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187290229344256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81407397 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187290481015808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198046059277312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169977952427248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189164939102720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169977935387888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8706775577192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83329687 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8731754010624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83329689 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119417015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49745416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31661981 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178684956985524 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8707004558276 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178684710965080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83329689 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178558709002240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8731520342016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187290229344256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81407397 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187290481015808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198046059277312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169977952427248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189164939102720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169977935387888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8706775577192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83329687 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8731754010624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83329689 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119417015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49745416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31661981 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178684956985524 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8707004558276 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178684710965080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83329689 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178558709002240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8731520342016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81424889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49756220 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119470077 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187331479407104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170015465745312 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 % _timestamp‚1.7885344789063008e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8733718049792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202791669 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189206623072768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8733718049792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8708963047470 trainer/global_step‚4399 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚359 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178598013042176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8708734066178 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚178597995039232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189206647012864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170015482784672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8883406258688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83347964 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5463.061353744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187331731091968 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202818041 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31668671 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198089732433920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8883109361152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202818041 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249{óšï&„éM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚178597995039232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189206647012864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170015482784672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8883406258688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83347964 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5463.061353744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187331731091968 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202818041 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31668671 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198089732433920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8883109361152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202818041 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81424889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49756220 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119470077 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187331479407104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170015465745312 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 % _timestamp‚1.7885344789063008e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8733718049792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202791669 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189206623072768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8733718049792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8708963047470 trainer/global_step‚4399 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚359 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178598013042176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8708734066178 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496çreÖ§2…- _wandbruntime‚5463  _runtime‚5463D„á$Z†Ô 0loss/FBt1/assignment_loss‚0.2162806152837523 1loss/SRqqt2/detection_loss‚2.8235934416453046 0loss/FRt3/assignment_loss‚1.3858542905913458 1loss/FBt4/assignment_loss‚0.23819265040153617 -loss/FRt2/detection_loss‚2.98826818381037 trainer/global_step‚4399 0loss/FRt1/assignment_loss‚2.9495278860247414 1loss/SRqqt1/assignment_loss‚1.791880127699936 1loss/SRqqt3/assignment_loss‚1.134581199905982 3loss/FBt3/assignment_loss‚2.096625855080453e-05 /loss/FRt1/detection_loss‚2.3334595186765803 /loss/FBt3/detection_loss‚0.7202385271320472 0loss/FRt2/assignment_loss‚3.2443039694002698 /loss/FRt4/assignment_loss‚3.361941389415575 0loss/FBt2/assignment_loss‚0.1853636769227857 /loss/FRt3/detection_loss‚1.4640121358780214 1loss/SRqqt1/detection_loss‚1.7489872662460102 .loss/FBt1/detection_loss‚2.562045814593633 .loss/FBt2/detection_loss‚2.955912172794342 _step‚360 /loss/FRt4/detection_loss‚3.2609824590061023 1loss/SRqqt3/detection_loss‚1.0604190701716825 %loss/total_loss‚43.25019722309948 epoch‚3 $ _timestamp‚1.788534478909788e+09 1loss/SRqqt4/detection_loss‚2.5950695872306824 1loss/SRqqt2/assignment_loss‚1.232686797529459 2loss/SRqqt4/assignment_loss‚1.6614179895037697 /loss/FBt4/detection_loss‚1.3351574869769993 _runtime‚5463.063380124êܺæ_‡Ù /loss/FRt4/detection_loss‚3.2609824590061023 1loss/SRqqt3/detection_loss‚1.0604190701716825 %loss/total_loss‚43.25019722309948 epoch‚3 $ _timestamp‚1.788534478909788e+09 1loss/SRqqt4/detection_loss‚2.5950695872306824 1loss/SRqqt2/assignment_loss‚1.232686797529459 2loss/SRqqt4/assignment_loss‚1.6614179895037697 /loss/FBt4/detection_loss‚1.3351574869769993 _runtime‚5463.063380124 0loss/FBt1/assignment_loss‚0.2162806152837523 1loss/SRqqt2/detection_loss‚2.8235934416453046 0loss/FRt3/assignment_loss‚1.3858542905913458 1loss/FBt4/assignment_loss‚0.23819265040153617 -loss/FRt2/detection_loss‚2.98826818381037 trainer/global_step‚4399 0loss/FRt1/assignment_loss‚2.9495278860247414 1loss/SRqqt1/assignment_loss‚1.791880127699936 1loss/SRqqt3/assignment_loss‚1.134581199905982 3loss/FBt3/assignment_loss‚2.096625855080453e-05 /loss/FRt1/detection_loss‚2.3334595186765803 /loss/FBt3/detection_loss‚0.7202385271320472 0loss/FRt2/assignment_loss‚3.2443039694002698 /loss/FRt4/assignment_loss‚3.361941389415575 0loss/FBt2/assignment_loss‚0.1853636769227857 /loss/FRt3/detection_loss‚1.4640121358780214 1loss/SRqqt1/detection_loss‚1.7489872662460102 .loss/FBt1/detection_loss‚2.562045814593633 .loss/FBt2/detection_loss‚2.955912172794342 _step‚360ètE¨ˆ‚(:„ Ú½ëÔØÐÓ› gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.715' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.543 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0btv‰‚(:ë Ú½ëÔØšúž network.sent‚ 1.0601112e+07 disk.dm-0.in‚21397.88671875 disk.drbd1260.out‚0.84375 disk.sda2.out‚0" proc.memory.rssMB‚ 3037.5234375 cpu‚25.30252484296194 network.recv‚ 1.802946e+06) disk./.usagePercent‚79.04096654067546$ disk.dm-0.out‚115870.76220703125 disk.sda.in‚0 disk.sda2.in‚0' disk.nvme0n1.out‚115870.76220703125 disk.sda3.in‚0$ memory_percent‚94.66018676757812 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 173.90625$ disk.nvme0c0n1.in‚21397.88671875) disk.nvme0c0n1.out‚115870.76220703125# proc.memory.availableMB‚874.875) proc.memory.percent‚18.53957176208496$ disk./.usageGB‚1177.767520904541 disk.dm-1.out‚ 173.90625 disk.drbd1260.in‚0" disk.nvme0n1.in‚21397.88671875 disk.sda3.out‚ 173.90625 proc.cpu.threads‚17‹º ‘Š‚(:† é½ëÔŚʕ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ÝXw~‹‚(:ó é½ëÔÏ눘 disk.sda1.out‚0 disk.sda.out‚ 174.34765625' disk.nvme0n1.out‚115961.59423828125" proc.memory.rssMB‚ 3037.5234375$ disk./.usageGB‚1177.744052886963 disk.dm-1.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚21397.88671875 disk.sda3.out‚ 174.34765625# proc.memory.availableMB‚874.625 cpu‚25.30204065886484 proc.cpu.threads‚17 disk.dm-0.in‚21397.88671875 disk.sda1.in‚0 disk.drbd1260.out‚0.84375 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0) disk.nvme0c0n1.out‚115961.59423828125 disk.dm-1.out‚ 174.34765625 disk.sda2.out‚0$ disk.nvme0c0n1.in‚21397.88671875$ memory_percent‚94.66171264648438) proc.memory.percent‚18.53957176208496 network.sent‚ 1.071002e+07 network.recv‚ 1.810055e+06) disk./.usagePercent‚79.03939158232481$ disk.dm-0.out‚115961.59423828125³´Œ‚(:„ ø½ëÔ—•¼– gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚127.32' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÂ*¦„‚(:ù ø½ëÔ岘 disk.dm-1.in‚0 disk.dm-1.out‚ 174.71484375 disk.sda1.out‚0$ memory_percent‚94.66021060943604( proc.memory.availableMB‚ 874.87109375 network.sent‚ 1.0716874e+07" disk.nvme0n1.in‚21397.88671875( disk.nvme0c0n1.out‚116470.3662109375" proc.memory.rssMB‚ 3037.5234375 cpu‚25.30144548333068 proc.cpu.threads‚17) disk./.usagePercent‚78.99584879687109% disk./.usageGB‚1177.0952339172363 disk.sda.in‚0 disk.sda.out‚ 174.71484375& disk.nvme0n1.out‚116470.3662109375 disk.dm-0.in‚21397.88671875 disk.sda1.in‚0 disk.drbd1260.in‚0 network.recv‚ 1.81366e+06 disk.sda2.out‚0 disk.sda3.out‚ 174.71484375$ disk.nvme0c0n1.in‚21397.88671875 disk.drbd1260.out‚ 0.84765625) proc.memory.percent‚18.53957176208496 disk.sda2.in‚0 disk.sda3.in‚0# disk.dm-0.out‚116470.3662109375_½Ž‚(:… ‡¾ëÔ ËÁ• gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.006' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6012 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~½£Ó‚(:ö ‡¾ëÔø¨¹— disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.84765625& disk.nvme0n1.out‚116705.4091796875$ memory_percent‚94.66018676757812 cpu‚25.300955490675832) disk./.usagePercent‚78.99427383852044% disk./.usageGB‚1177.0717658996582$ disk.nvme0c0n1.in‚21397.88671875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.in‚0# proc.memory.availableMB‚874.875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.in‚21397.88671875 disk.sda3.out‚ 175.12890625" proc.memory.rssMB‚ 3037.5234375( disk.nvme0c0n1.out‚116705.4091796875 disk.dm-0.in‚21397.88671875# disk.dm-0.out‚116705.4091796875 disk.dm-1.out‚ 175.12890625 disk.sda.out‚ 175.12890625 disk.sda2.out‚0 disk.sda3.in‚0) proc.memory.percent‚18.53957176208496 network.sent‚ 1.0721081e+07 network.recv‚ 1.818269e+06«ó¶2- _wandbruntime‚5520  _runtime‚5520þú\ÃÆ‘À & lr-AdamW/pg1‚0.0006506133313831462 & lr-AdamW/pg2‚0.0006506133313831462 trainer/global_step‚4449 $ _timestamp‚1.788534536781944e+09 _runtime‚5520.935471971 _step‚361¬â£FË’Å _step‚361 & lr-AdamW/pg1‚0.0006506133313831462 & lr-AdamW/pg2‚0.0006506133313831462 trainer/global_step‚4449 $ _timestamp‚1.788534536781944e+09 _runtime‚5520.935471971éß„„ü2“- _wandbruntime‚5520  _runtime‚5520/pß'”ÙO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120754967 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191249137605120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8829736048640 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120781311 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180523028854784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120754967 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180659412897232 _step‚362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32011642 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8980950790144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200230407149056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180523028854784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8829969717248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82297158 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191249161545216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204998404 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189352998572032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84243437 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50285516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171854453259088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189352746900480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180523010851840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84243439 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8829736048640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180659166876788 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8804713617700 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180523010851840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8981245603840 $ _timestamp‚1.788534536783162e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189352746900480 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8980950790144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200230407149056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180523028854784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8829969717248 M@DeviceStatsMonitor.¥Kí~on_train_batch_start/inactive_split.all.freed‚82297158 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191249161545216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204998404 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189352998572032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84243437 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50285516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171854453259088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189352746900480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180523010851840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84243439 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8829736048640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180659166876788 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8804713617700 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180523010851840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8981245603840 $ _timestamp‚1.788534536783162e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189352746900480 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180562296888832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191290821575168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180562296888832 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200274079178240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84261712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84261712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82314970 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8831700074496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205069743 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205043371 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8831700074496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180562314891776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205069743 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84261714 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82315613 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120808029 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180562314891776 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚ 5522.41324465 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189393996963328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171891983616512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189394248648192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171892000655872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205043371 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50296320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8983233662976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8831933756416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84261714 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180562296888832 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200274079178240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84261712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84261712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82314970 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8831700074496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205069743 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205043371 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8831700074496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180562314891776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205069743 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84261714 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82315613 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120808029 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180562314891776 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚ 5522.41324465 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189393996963328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171891983616512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189394248648192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171892000655872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205043371 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50296320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8983233662976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8831933756416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84261714 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180562296888832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191290821575168ë¦b¼2™- _wandbruntime‚5522  _runtime‚5522zÞs–[šÕ 0loss/FRt3/assignment_loss‚1.1827866312785027 2loss/SRqqt4/assignment_loss‚1.9419611127903829 1loss/FBt4/assignment_loss‚0.19143424490924746 .loss/FRt4/detection_loss‚3.015108954906464 /loss/FBt4/detection_loss‚1.1432276193776891 0loss/FRt2/assignment_loss‚3.8119004257023335 0loss/FRt4/assignment_loss‚3.0749829745292665 3loss/FBt3/assignment_loss‚0.0034894356901219886 %loss/total_loss‚43.74339099005901 1loss/SRqqt2/assignment_loss‚2.109461839000384 1loss/SRqqt1/detection_loss‚1.5719234238971362 0loss/SRqqt4/detection_loss‚2.590269617163218 trainer/global_step‚4449 $ _timestamp‚1.788534538262236e+09 _step‚364 2loss/SRqqt1/assignment_loss‚1.6246687752364033 .loss/FBt1/detection_loss‚2.161970364696839 .loss/FBt2/detection_loss‚2.457964289646882 _runtime‚5522.415675272 1loss/FBt1/assignment_loss‚0.19764292024652766 0loss/FBt2/assignment_loss‚0.2855585453705946 /loss/FRt1/detection_loss‚2.6080851870424606 1loss/SRqqt3/detection_loss‚1.1085805114901934 /loss/FRt1/assignment_loss‚4.117222395013361 .loss/FRt2/detection_loss‚2.967024715617299 /loss/FRt3/detection_loss‚1.5594593500479674 1loss/SRqqt2/detection_loss‚2.4182746450106305 2loss/SRqqt3/assignment_loss‚0.8646077711309772 .loss/FBt3/detection_loss‚0.735785240264133 epoch‚3ëŠ)t`›Ú /loss/FRt1/assignment_loss‚4.117222395013361 .loss/FRt2/detection_loss‚2.967024715617299 /loss/FRt3/detection_loss‚1.5594593500479674 1loss/SRqqt2/detection_loss‚2.4182746450106305 2loss/SRqqt3/assignment_loss‚0.8646077711309772 .loss/FBt3/detection_loss‚0.735785240264133 epoch‚3 0loss/FRt3/assignment_loss‚1.1827866312785027 2loss/SRqqt4/assignment_loss‚1.9419611127903829 1loss/FBt4/assignment_loss‚0.19143424490924746 .loss/FRt4/detection_loss‚3.015108954906464 /loss/FBt4/detection_loss‚1.1432276193776891 0loss/FRt2/assignment_loss‚3.8119004257023335 0loss/FRt4/assignment_loss‚3.0749829745292665 3loss/FBt3/assignment_loss‚0.0034894356901219886 %loss/total_loss‚43.74339099005901 1loss/SRqqt2/assignment_loss‚2.109461839000384 1loss/SRqqt1/detection_loss‚1.5719234238971362 0loss/SRqqt4/detection_loss‚2.590269617163218 trainer/global_step‚4449 $ _timestamp‚1.788534538262236e+09 _step‚364 2loss/SRqqt1/assignment_loss‚1.6246687752364033 .loss/FBt1/detection_loss‚2.161970364696839 .loss/FBt2/detection_loss‚2.457964289646882 _runtime‚5522.415675272 1loss/FBt1/assignment_loss‚0.19764292024652766 0loss/FBt2/assignment_loss‚0.2855585453705946 /loss/FRt1/detection_loss‚2.6080851870424606 1loss/SRqqt3/detection_loss‚1.1085805114901934ì²Å œœ‚(:‘ –¾ëÔê‘Þ˜ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.362' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.272399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×'á‰}‚(:ò –¾ëÔäÞ…œ# disk.nvme0c0n1.in‚ 21532.4609375( disk.nvme0c0n1.out‚116874.1826171875 disk.dm-0.in‚ 21532.4609375# disk.dm-0.out‚116874.1826171875 disk.sda.out‚ 175.6484375 network.sent‚ 1.0725964e+07 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 175.6484375$ memory_percent‚94.65887546539307( proc.memory.availableMB‚ 875.08984375) proc.memory.percent‚18.53957176208496 cpu‚25.30037092599274 disk.dm-1.out‚ 175.6484375 network.recv‚ 1.821543e+06 disk.dm-1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 21532.4609375 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0& disk.nvme0n1.out‚116874.1826171875" proc.memory.rssMB‚ 3037.5234375 disk.drbd1260.out‚ 0.8515625) disk./.usagePercent‚78.99584854086356$ disk./.usageGB‚1177.095230102539"q3È‘ž‚(:† ¥¾ëÔÏ·¢• gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4244 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·…2—Ÿ‚(:ó ¥¾ëÔ¿åé— disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 176.1953125 disk.sda2.out‚0# disk.dÿ7#çm-0.out‚117305.6123046875 disk.drbd1260.out‚ 0.8515625( proc.memory.availableMB‚ 875.08203125" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 disk.sda3.out‚ 176.1953125# disk.nvme0c0n1.in‚ 21532.4609375! disk.nvme0n1.in‚ 21532.4609375& disk.nvme0n1.out‚117305.6123046875 disk.sda3.in‚0 disk.dm-1.out‚ 176.1953125$ memory_percent‚94.65892314910889 cpu‚25.29991747640387 disk.sda.in‚0 disk.dm-0.in‚ 21532.4609375 disk.dm-1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚117305.6123046875 network.sent‚ 1.0835154e+07 network.recv‚ 1.828063e+06) disk./.usagePercent‚78.99584726082588% disk./.usageGB‚1177.0952110290527Ypôœœ ‚(:‘ ´¾ëÔ©Ü× gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.077' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.415399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥~^‚¡‚(:÷ ´¾ëÔÙÛþ› disk.sda.in‚0! disk.nvme0n1.in‚ 21532.4609375 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 21532.4609375( disk.nvme0c0n1.out‚117493.0029296875 disk.sda2.out‚0 disk.drbd1260.in‚0# disk.dm-0.out‚117493.0029296875 cpu‚25.299388572232765 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1260.out‚ 0.85546875& disk.nvme0n1.out‚117493.0029296875 disk.dm-0.in‚ 21532.4609375# memory_percent‚94.6605920791626( proc.memory.availableMB‚ 874.80859375" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 network.sent‚ 1.0840314e+07) disk./.usagePercent‚78.99584316470533% disk./.usageGB‚1177.0951499938965 disk.dm-1.out‚ 176.62890625 disk.sda1.in‚0 disk.sda.out‚ 176.62890625 disk.sda2.in‚0 disk.sda3.out‚ 176.62890625 network.recv‚ 1.831976e+06 disk.dm-1.in‚0 Ræî‘¢‚(:† þëÔºƒÁš gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2604 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 üù{£‚(:ð þëÔçËÒŸ! disk.nvme0n1.in‚ 21532.4609375& disk.nvme0n1.out‚117641.5732421875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 21532.4609375# memory_percent‚94.6605920791626 disk.dm-1.out‚ 177.265625) proc.memory.percent‚18.53957176208496 cpu‚25.29891059326397 proc.cpu.threads‚17 disk.sda.out‚ 177.265625 disk.sda2.in‚0 network.sent‚ 1.0846481e+07 network.recv‚ 1.836643e+06) disk./.usagePercent‚78.99584316470533% disk./.usageGB‚1177.0951499938965 disk.drbd1260.out‚ 0.85546875 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 177.265625# disk.nvme0c0n1.in‚ 21532.4609375( disk.nvme0c0n1.out‚117641.5732421875# disk.dm-0.out‚117641.5732421875 disk.dm-1.in‚0( proc.memory.availableMB‚ 874.80859375" proc.memory.rssMB‚ 3037.5234375 disk.sda2.out‚0ÃÐF‹2¤- _wandbruntime‚5582  _runtime‚5582&¨ Å¥¿ _runtime‚5582.719318077 _step‚365 % lr-AdamW/pg1‚0.000645844037387588 % lr-AdamW/pg2‚0.000645844037387588 trainer/global_step‚4499 % _timestamp‚1.7885345985657346e+09«ö½žÊ¦Ä _step‚365 % lr-AdamW/pg1‚0.000645844037387588 % lr-AdamW/pg2‚0.000645844037387588 trainer/global_step‚4499 % _timestamp‚1.7885345985657346e+09 _runtime‚5582.719318077í·§ßÁ2§- _wandbruntime‚5582  _runtime‚5582P©Õ£à'¨ÚO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182487312701440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122119263 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182487330704384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202414392636416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122092919 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83188274 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193333360047616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚366 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85157187 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85157189 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32361975 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173730971130288 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122092919 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8902651658208 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191415516128256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191415264456704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8902880639292 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207276452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9081032588800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885345985668335e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85157187 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8928185423872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191415516128256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207250106 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191415264456704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83187591 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182633622788496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32362656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50825616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777þ»Nå'©ßO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182487330704384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202414392636416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122092919 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83188274 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193333360047616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚366 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85157187 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85157189 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32361975 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173730971130288 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122092919 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8902651658208 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191415516128256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191415264456704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8902880639292 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207276452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9081032588800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885345985668335e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85157187 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8928185423872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191415516128256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207250106 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191415264456704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83187591 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182633622788496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32362656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50825616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182487312701440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122119263 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104îåXó­2ª- _wandbruntime‚5583  _runtime‚5583¢ƒê «äM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193375020077568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚4499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191456766204416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207321445 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173768518527072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207295073 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182526598738432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191456514519552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32368609 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122145981 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202458071907328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182673111634906 _runtime‚5583.988271037 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8904839128486 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85175464 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122145981 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83205684 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207321445 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191456766204416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207295073 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83205027 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/rLº­¤@eserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚4499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191456766204416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207321445 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173768518527072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207295073 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182526598738432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191456514519552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32368609 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122145981 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202458071907328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182673111634906 _runtime‚5583.988271037 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8904839128486 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85175464 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122145981 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83205684 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207321445 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191456766204416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207295073 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83205027 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ï™™·2­- _wandbruntime‚5583  _runtime‚5583‘7€B[®Õ .loss/FBt4/detection_loss‚1.107443489689142 .loss/FBt2/detection_loss‚2.816066318324634 0loss/FRt4/assignment_loss‚3.1115832388401032 2loss/SRqqt3/assignment_loss‚1.1872978978139206 2loss/SRqqt4/assignment_loss‚2.2865451803574195 1loss/FBt1/assignment_loss‚0.29084066731133507 -loss/FRt1/detection_loss‚2.17701174282475 -loss/FRt2/detection_loss‚3.13100278267154 % _timestamp‚1.7885345998364503e+09 2loss/SRqqt1/assignment_loss‚1.4313624826846298 1loss/FBt2/assignment_loss‚0.22062960279884333 .loss/FRt3/detection_loss‚1.764114489700173 2loss/SRqqt2/assignment_loss‚1.5917861452626605 2loss/FBt3/assignment_loss‚0.012071563877801358 1loss/SRqqt1/detection_loss‚1.7780609197086759 /loss/FBt3/detection_loss‚0.9052655373622353 %loss/total_loss‚43.88655941683351 epoch‚3 /loss/FRt2/assignment_loss‚3.197550486635279 .loss/FRt4/detection_loss‚3.284321701526642 1loss/SRqqt3/detection_loss‚0.9761966084587878 /loss/FBt1/detection_loss‚2.3729238729727897 0loss/FRt1/assignment_loss‚3.7097250397612407 0loss/FRt3/assignment_loss‚1.4544797388600632 1loss/SRqqt2/detection_loss‚2.4498793848099245 1loss/SRqqt4/detection_loss‚2.3353679340619307 _step‚368 1loss/FBt4/assignment_loss‚0.29503259051898084 _runtime‚5583.989803916 trainer/global_step‚4499ÀÏã`¯Ú trainer/global_step‚4499 .loss/FBt4/detection_loss‚1.107443489689142 .loss/FBt2/detection_loss‚2.816066318324634 0loss/FRt4/assignment_loss‚3.1115832388401032 2loss/SRqqt3/assignment_loss‚1.1872978978139206 2loss/SRqqt4/assignment_loss‚2.2865451803574195 1loss/FBt1/assignment_loss‚0.29084066731133507 -loss/FRt1/detection_loss‚2.17701174282475 -loss/FRt2/detection_loss‚3.13100278267154 % _timestamp‚1.7885345998364503e+09 2loss/SRqqt1/assignment_loss‚1.4313624826846298 1loss/FBt2/assignment_loss‚0.22062960279884333 .loss/FRt3/detection_loss‚1.764114489700173 2loss/SRqqt2/assignment_loss‚1.5917861452626605 2loss/FBt3/assignment_loss‚0.012071563877801358 1loss/SRqqt1/detection_loss‚1.7780609197086759 /loss/FBt3/detection_loss‚0.9052655373622353 %loss/total_loss‚43.88655941683351 epoch‚3 /loss/FRt2/assignment_loss‚3.197550486635279 .loss/FRt4/detection_loss‚3.284321701526642 1loss/SRqqt3/detection_loss‚0.9761966084587878 /loss/FBt1/detection_loss‚2.3729238729727897 0loss/FRt1/assignment_loss‚3.7097250397612407 0loss/FRt3/assignment_loss‚1.4544797388600632 1loss/SRqqt2/detection_loss‚2.4498793848099245 1loss/SRqqt4/detection_loss‚2.3353679340619307 _step‚368 1loss/FBt4/assignment_loss‚0.29503259051898084 _runtime‚5583.989803916ðŒ¬½°‚(:… Ò¾ëÔ‡üª— gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0932 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%¯€±‚(:õ Ò¾ëÔ´ÙÖ—) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 178.0078125 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚117902.3916015625 cpu‚25.29849888121843 network.recv‚ 1.839931e+06$ disk.nvme0c0n1.in‚21724.28515625( disk.nvme0c0n1.out‚117902.3916015625 disk.sda1.in‚0 disk.drbd1260.out‚0.859375 disk.sda.in‚0 network.sent‚ 1.0851336e+07) disk./.usagePercent‚78.98703061735418% disk./.usageGB‚1176.9638366699219 disk.dm-0.in‚21724.28515625# disk.dm-0.out‚117902.3916015625" disk.nvme0n1.in‚21724.28515625 disk.sda3.in‚0" proc.memory.rssMB‚ 3037.5234375 disk.dm-1.in‚0 disk.dm-1.out‚ 178.0078125 disk.sda3.out‚ 178.0078125$ memory_percent‚94.65739727020264( proc.memory.availableMB‚ 875.33203125§ë3u‘²‚(:† á¾ëÔ¾°š– gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*Zår³‚(:ç á¾ëÔè㌚ disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0) disk./.usagePercent‚78.98701397686445% disk./.usageGB‚1176.9635887145996 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚118231.1806640625" proc.memory.rssMB‚ 3037.5234375 cpu‚25.297948864976572 disk.sda3.in‚0 disk.dm-0.in‚21724.28515625 disk.dm-1.out‚179.125 disk.drbd1260.out‚0.859375# memory_percent‚94.6589469909668 proc.cpu.threads‚17" disk.nvme0n1.in‚21724.28515625 disk.sda3.out‚179.125# disk.dm-0.out‚118231.1806640625 disk.dm-1.in‚0 disk.sda.out‚179.125& proc.memory.availableMB‚ 875.078125) proc.memory.percent‚18.53957176208496 network.sent‚ 1.0960363e+07 network.recv‚ 1.847153e+06& disk.nvme0n1.out‚118231.1806640625$ disk.nvme0c0n1.in‚21724.28515625 disk.sda1.in‚0"3e4‘´‚(:† ð¾ëÔµö­– gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.827' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5654 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿î¸†µ‚(:û ð¾ëÔ®œý™ disk.sda.in‚0 disk.sda.out‚ 179.78515625 disk.sda3.in‚0 disk.sda3.out‚ 179.78515625 disk.dm-1.out‚ 179.78515625 disk.drbd1260.out‚ 0.86328125 network.sent‚ 1.096538e+07% disk./.usageGB‚1177.1526794433594 disk.sda2.out‚0" disk.nvme0n1.in‚21724.28515625 disk.sda1.out‚0" proc.memory.rssMB‚ 3037.5234375 cpu‚25.29746968626468) disk./.usagePercent‚78.99970401432921$ disk.nvme0c0n1.in‚21724.28515625) disk.nvme0c0n1.out‚118611.18798828125 disk.dm-0.in‚21724.28515625$ disk.dm-0.out‚118611.18798828125 disk.drbd1260.in‚0# memory_percent‚94.6589708328247 proc.cpu.threads‚17 network.recv‚ 1.850366e+06 disk.sda2.in‚0' disk.nvme0n1.out‚118611.18798828125 disk.dm-1.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 875.07421875) proc.memory.percent‚18.53957176208496ë´<}‘¶‚(:† ÿ¾ëÔ¬ŽÚ— gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽDãv·‚(:ë ÿ¾ëÔû½±› disk.sda1.out‚0) disk.nvme0c0n1.out‚118978.84033203125( proc.memory.availableMB‚ 874.82421875) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17 network.sent‚ 1.0971423e+07 disk.dm-1.in‚0 disk.dm-1.out‚180.375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.86328125 disk.sda.out‚180.375 disk.sda2.out‚0' disk.nvme0n1.out‚118978.84033203125% disk./.usageGB‚1177.1523323059082 disk.sda.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚21724.28515625 disk.sda3.in‚0 disk.sda3.out‚180.375$ disk.dm-0.out‚118978.84033203125" proc.memory.rssMB‚ 3037.5234375 network.recv‚ 1.855e+06( disk./.usagePercent‚78.9996807176436 disk.sda1.in‚0$ disk.nvme0c0n1.in‚21724.28515625 disk.dm-0.in‚21724.28515625$ memory_percent‚94.66049671173096 cpu‚25.296979131940446' P2¸- _wandbruntime‚5643  _runtime‚5643ú8˜µÇ¹Á & lr-AdamW/pg1‚0.0006410252407495569 & lr-AdamW/pg2‚0.0006410252407495569 trainer/global_step‚4549 % _timestamp‚1.7885346596949866e+09 _runtime‚5643.848588998 _step‚369%„ëwÌºÆ & lr-AdamW/pg1‚0.0006410252407495569 & lr-AdamW/pg2‚0.0006410252407495569 trainer/global_step‚4549 % _timestamp‚1.7885346596949866e+09 _runtime‚5643.848588998 _step‚369ñSú”2»- _wandbruntime‚5643  _runtime‚5643èXaà'¼ÚO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204598103025664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86070939 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚370 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184451614551040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123430871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 329239552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86070937 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123457215 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184451614551040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚184608324720648 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32705716 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9000589698716 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51365718 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86070937 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84072134 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123457215 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86070939 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 % _timestamp‚1.7885346596965294e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193477782012928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209528154 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204598432265216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123430871 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193478033684480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚184608078700204 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚4549 _runtime‚5643.850459264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9180873715200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 305299456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175607506040848 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193478033684480 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792MÍqW ½ßO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86070939 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚370 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184451614551040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123430871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 329239552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86070937 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123457215 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184451614551040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚184608324720648 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32705716 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9000589698716 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51365718 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 WIDevice™ªB©ŽStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86070937 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84072134 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123457215 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86070939 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 % _timestamp‚1.7885346596965294e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193477782012928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209528154 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204598432265216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123430871 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193478033684480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚184608078700204 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚4549 _runtime‚5643.850459264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9180873715200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 305299456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175607506040848 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193478033684480 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204598103025664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496ò/[¨2¾- _wandbruntime‚5645  _runtime‚5645úEpê&¿äM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9002548187702 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123483933 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32713434 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9028365169664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9028365169664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5645.152954807 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51376520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193519032075776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32712784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 trainer/global_step‚4549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184490918590976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86089212 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84089302 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175645036398272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885346609986258e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209573147 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚184647567546614 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195459218579968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9182857807360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9002777168994 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193519283760640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204642100327424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86089214 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9028131487744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184490918590976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209573147 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209546775 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚371 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123457563 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84089954 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9182567201280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51376518 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9028131487744 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5645.152954807 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51376520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193519032075776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32712784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 trainer/global_step‚4549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184490918590976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86089212 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84089302 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175645036398272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885346609986258e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209573147 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚184647567546614 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195459218579968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9182857807360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9002777168994 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193519283760640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204642100327424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86089214 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9028131487744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184490918590976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209573147 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209546775 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚371 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123457563 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84089954 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9182567201280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51376518 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9028131487744 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86984687 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86984689 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚186582534611912 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86984689 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9124616837120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197501757052416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177484006872688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9098527739224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9124383168512 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9124616837120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51905816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33054867 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197501733112320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195540551240704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84960685 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206782118504960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9280385392640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177484023912048 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚374 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195540299569152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124768823Ë{Cä'ÑÞO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚186415916400640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195540299569152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124768823 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86984687 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86984689 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚186582534611912 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86984689 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9124616837120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197501757052416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177484006872688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9098527739224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9124383168512 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9124616837120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51905816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33054867 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197501733112320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195540551240704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84960685 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206782118504960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9280385392640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177484023912048 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚374 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9282669248000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9126580876288 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9126347194368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211798477 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84978009 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211824849 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9282374447616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51916620 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33061389 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211798477 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124795515 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚186622023458322 _runtime‚5705.796233512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33060687 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206826110270464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9126347194368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87002962 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚186455202437632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186455220440576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206825791529984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚4599 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211824849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124821885 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885347216415553e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84977305q<ï&ÔéM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9126580876288 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211824849 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9282374447616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51916620 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33061389 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211798477 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124795515 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚186622023458322 _runtime‚5705.796233512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33060687 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206826110270464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9126347194368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87002962 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚186455202437632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186455220440576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206825791529984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚4599 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211824849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124821885 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885347216415553e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84977305 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197543441022464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87002964 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚186455202437632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87002964 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197543417082368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177521554269472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚195581549632000 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124795515 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9282669248000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9126580876288 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9126347194368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211798477 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84978009 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E_[ÖÕ .loss/FRt4/detection_loss‚2.923833312988281 1loss/SRqqt3/detection_loss‚0.9763037743115867 1loss/SRqqt4/detection_loss‚2.5844785451889036 /loss/FBt3/detection_loss‚0.6069214947691018 0loss/FRt1/assignment_loss‚2.9047715976124717 %loss/total_loss‚42.51655426716949 $ _timestamp‚1.788534721646045e+09 _step‚376 /loss/FRt2/assignment_loss‚3.682362349137016 2loss/SRqqt3/assignment_loss‚0.9999382526542898 1loss/FBt1/assignment_loss‚0.27254395542450616 0loss/SRqqt1/detection_loss‚2.079730361700058 .loss/FBt1/detection_loss‚2.302579484383265 /loss/FRt3/assignment_loss‚1.232136934190183 4loss/FBt3/assignment_loss‚2.3156275904929142e-05 /loss/FRt1/detection_loss‚2.2476090639829636 .loss/FRt2/detection_loss‚3.316934005073879 .loss/FRt3/detection_loss‚1.488456398349356 /loss/FBt2/detection_loss‚1.8710762384262951 1loss/SRqqt1/assignment_loss‚1.409958056277699 1loss/SRqqt2/assignment_loss‚1.734363262852033 2loss/SRqqt4/assignment_loss‚1.6888567373156547 0loss/SRqqt2/detection_loss‚2.824035962422689 epoch‚3 trainer/global_step‚4599 /loss/FRt4/assignment_loss‚2.898033046722412 2loss/FBt4/assignment_loss‚0.047326414957069574 /loss/FBt4/detection_loss‚2.1681884076784956 _runtime‚5705.799550585 1loss/FBt2/assignment_loss‚0.25609345447537635„’u’`×Ú 2loss/SRqqt3/assignment_loss‚0.9999382526542898 1loss/FBt1/assignment_loss‚0.27254395542450616 0loss/SRqqt1/detection_loss‚2.079730361700058 .loss/FBt1/detection_loss‚2.302579484383265 /loss/FRt3/assignment_loss‚1.232136934190183 4loss/FBt3/assignment_loss‚2.3156275904929142e-05 /loss/FRt1/detection_loss‚2.2476090639829636 .loss/FRt2/detection_loss‚3.316934005073879 .loss/FRt3/detection_loss‚1.488456398349356 /loss/FBt2/detection_loss‚1.8710762384262951 1loss/SRqqt1/assignment_loss‚1.409958056277699 1loss/SRqqt2/assignment_loss‚1.734363262852033 2loss/SRqqt4/assignment_loss‚1.6888567373156547 0loss/SRqqt2/detection_loss‚2.824035962422689 epoch‚3 trainer/global_step‚4599 /loss/FRt4/assignment_loss‚2.898033046722412 2loss/FBt4/assignment_loss‚0.047326414957069574 /loss/FBt4/detection_loss‚2.1681884076784956 _runtime‚5705.799550585 1loss/FBt2/assignment_loss‚0.25609345447537635 .loss/FRt4/detection_loss‚2.923833312988281 1loss/SRqqt3/detection_loss‚0.9763037743115867 1loss/SRqqt4/detection_loss‚2.5844785451889036 /loss/FBt3/detection_loss‚0.6069214947691018 0loss/FRt1/assignment_loss‚2.9047715976124717 %loss/total_loss‚42.51655426716949 $ _timestamp‚1.788534721646045e+09 _step‚376 /loss/FRt2/assignment_loss‚3.682362349137016ø|ÜwȜ؂(:‘ Ê¿ëÔò„— gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.520599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í–7rÙ‚(:ç Ê¿ëÔäÙ„™ cpu‚25.294735798233674 proc.cpu.threads‚17& disk.nvme0n1.out‚120232.9130859375 network.sent‚ 1.1101364e+07 network.recv‚ 1.876401e+06 disk.sda2.in‚0 disk.dm-0.in‚21861.43359375% proc.memory.availableMB‚ 874.71875 disk.drbd1260.out‚0.875 disk.sda.out‚182.6875 disk.sda3.in‚0$ disk.nvme0c0n1.in‚21861.43359375( disk.nvme0c0n1.out‚120232.9130859375" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496) disk./.usagePercent‚79.00430037359897% disk./.usageGB‚1177.2211685180664 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0# disk.dm-0.out‚120232.9130859375 disk.dm-1.in‚0 disk.dm-1.out‚182.6875 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚21861.43359375 disk.sda3.out‚182.6875$ memory_percent‚94.66114044189453ï²7‘Ú‚(:† Ù¿ëÔԇ̖ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9546 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l†ŠlÛ‚(:á Ù¿ëÔ¨¸——% disk./.usageGB‚1177.2209014892578 disk.nvme0c0n1.in‚ 22029.6875# disk.dm-0.out‚120608.0849609375 disk.sda.in‚0 disk.nvme0n1.in‚ 22029.6875 proc.cpu.threads‚17 disk.dm-1.in‚0& disk.nvme0n1.out‚120608.0849609375 disk.sda3.out‚ 183.34375$ memory_percent‚94.66044902801514 cpu‚25.294246240760735 disk.drbd1260.out‚0.875 network.recv‚ 1.883251e+06) disk./.usagePercent‚79.00428245307157( disk.nvme0c0n1.out‚120608.0849609375 disk.dm-1.out‚ 183.34375( proc.memory.availableMB‚ 874.83203125" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 disk.dm-0.in‚ 22029.6875 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 183.34375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 network.sent‚ 1.1210523e+07Õ֜܂(:‘ è¿ëÔµíÈ— gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.022' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.604399999999995 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÔ¸ }Ý‚(:ò è¿ëÔåÎÄš disk.drbd1260.out‚ 0.87890625 disk.sda2.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 874.58984375" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496% disk./.usageGB‚1177.2834014892578 disk.dm-0.in‚ 22029.6875 disk.dm-1.out‚ 183.79296875 disk.sda2.out‚0 disk.nvme0n1.in‚ 22029.6875 disk.sda3.out‚ 183.79296875 cpu‚25.293798466596034 network.recv‚ 1.886478e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.nvme0c0n1.in‚ 22029.6875$ memory_percent‚94.66192722320557 proc.cpu.threads‚17 network.sent‚ 1.1215361e+07) disk./.usagePercent‚79.00847688051256 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 183.79296875' disk.nvme0n1.out‚120761.86669921875) disk.nvme0c0n1.out‚120761.86669921875$ disk.dm-0.out‚120761.86669921875µÛ|4‘Þ‚(:† ÷¿ëÔï“Á• gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6662 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 0&:{ß‚(:ð ÷¿ëÔ»þ½™ disk.dm-1.out‚ 184.16015625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.87890625' disk.nvme0n1.out‚121102.68310546875) disk.nvme0c0n1.out‚121102.68310546875$ memory_percent‚94.66347694396973' proc.memory.availableMB‚ 874.3359375" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 network.recv‚ 1.891275e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 22029.6875 disk.drbd1260.in‚0 cpu‚25.29326328881527 proc.cpu.threads‚17 network.sent‚ 1.1221613e+07) disk./.usagePercent‚79.01690029641166% disk./.usageGB‚1177.4089164733887 disk.sda.out‚ 184.16015625 disk.nvme0n1.in‚ 22029.6875$ disk.dm-0.out‚121102.68310546875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 184.16015625 disk.dm-0.in‚ 22029.6875KÙ2à- _wandbruntime‚5764  _runtime‚5764ôDüÇáÁ % _timestamp‚1.7885347798646927e+09 _runtime‚5764.018280084 _step‚377 & lr-AdamW/pg1‚0.0006312431665730594 & lr-AdamW/pg2‚0.0006312431665730594 trainer/global_step‚4649|g»ñÌâÆ & lr-AdamW/pg2‚0.0006312431665730594 trainer/global_step‚4649 % _timestamp‚1.7885347798646927e+09 _runtime‚5764.018280084 _step‚377 & lr-AdamW/pg1‚0.0006312431665730594ù?n˜“2ã- _wandbruntime‚5764  _runtime‚5764×&úûà'äÚO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9196694760816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197603068796928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33400029 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179360541783248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188380236253184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚4649 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188380218250240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9222832543744 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126106775 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52445916 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188557236544064 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87898439 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85845297 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179360524743888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197602817125376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126133119 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9196465779732 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87898439 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197602817125376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85845947 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9380554914816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197603068796928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9222832543744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214031558 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33399381 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5764.019763401 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126106775 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9380266392576 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/aVBB –llocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33400029 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179360541783248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188380236253184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚4649 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188380218250240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9222832543744 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126106775 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52445916 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188557236544064 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87898439 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85845297 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179360524743888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197602817125376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126133119 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9196465779732 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87898439 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197602817125376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85845947 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9380554914816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197603068796928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9222832543744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214031558 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33399381 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5764.019763401 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126106775 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9380266392576 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9224796582912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52456720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885347813236015e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214076551 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87916714 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87916714 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85862762 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179398055101312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197644067188224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33406698 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188596479370030 trainer/global_step‚4649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9224562900992 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214050179 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52456718 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126133467 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188419522290176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9224796582912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214050179 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179398072140672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188419504287232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9382263476224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33406044 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209009879060992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126159837 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87916712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126133467 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214076551 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87916712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126159837 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199627639524864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188596725390682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209010189412864—¢Ðoï&èéM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214076551 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87916712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126159837 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199627639524864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188596725390682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209010189412864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9224562900992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85863418 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9224796582912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52456720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885347813236015e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214076551 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87916714 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87916714 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85862762 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179398055101312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197644067188224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33406698 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188596479370030 trainer/global_step‚4649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9224562900992 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214050179 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52456718 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126133467 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188419522290176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9224796582912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214050179 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179398072140672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188419504287232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9382263476224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33406044 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209009879060992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126159837 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87916712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126133467 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496û–PÞå2é- _wandbruntime‚5765  _runtime‚5765sNO \êÖ /loss/FRt1/detection_loss‚2.1008112431579846 /loss/FRt3/detection_loss‚1.4964983473355706 epoch‚3 trainer/global_step‚4649 % _timestamp‚1.7885347813268762e+09 0loss/FBt3/detection_loss‚0.38895870708963937 2loss/SRqqt2/assignment_loss‚1.9499429035931826 2loss/SRqqt4/assignment_loss‚1.7785410409712272 0loss/FBt1/assignment_loss‚0.3726630356386035 2loss/FBt3/assignment_loss‚0.003117367185691617 1loss/SRqqt3/detection_loss‚0.9584778357725755 -loss/FBt1/detection_loss‚2.61909875699452 /loss/FBt2/detection_loss‚2.4126830661998078 /loss/FBt4/detection_loss‚1.6218881612193996 0loss/FRt4/assignment_loss‚2.5748564158167158 0loss/FBt2/assignment_loss‚0.5074509236017761 _runtime‚5765.480337366 2loss/SRqqt3/assignment_loss‚1.1853514373368395 /loss/FRt2/detection_loss‚2.8265759425778545 &loss/total_loss‚44.824608931564654 0loss/FBt4/assignment_loss‚0.4187019027209191 /loss/FRt3/assignment_loss‚1.534528468106244 .loss/FRt4/detection_loss‚2.860421940258571 0loss/SRqqt4/detection_loss‚2.780926595563474 /loss/FRt1/assignment_loss‚4.456896888904082 /loss/FRt2/assignment_loss‚3.815487984688051 _step‚380 2loss/SRqqt1/assignment_loss‚1.5259882579698707 1loss/SRqqt1/detection_loss‚2.0549620417031376 1loss/SRqqt2/detection_loss‚2.5797796671589217cO*aëÛ 2loss/SRqqt3/assignment_loss‚1.1853514373368395 /loss/FRt2/detection_loss‚2.8265759425778545 &loss/total_loss‚44.824608931564654 0loss/FBt4/assignment_loss‚0.4187019027209191 /loss/FRt3/assignment_loss‚1.534528468106244 .loss/FRt4/detection_loss‚2.860421940258571 0loss/SRqqt4/detection_loss‚2.780926595563474 /loss/FRt1/assignment_loss‚4.456896888904082 /loss/FRt2/assignment_loss‚3.815487984688051 _step‚380 2loss/SRqqt1/assignment_loss‚1.5259882579698707 1loss/SRqqt1/detection_loss‚2.0549620417031376 1loss/SRqqt2/detection_loss‚2.5797796671589217 /loss/FRt1/detection_loss‚2.1008112431579846 /loss/FRt3/detection_loss‚1.4964983473355706 epoch‚3 trainer/global_step‚4649 % _timestamp‚1.7885347813268762e+09 0loss/FBt3/detection_loss‚0.38895870708963937 2loss/SRqqt2/assignment_loss‚1.9499429035931826 2loss/SRqqt4/assignment_loss‚1.7785410409712272 0loss/FBt1/assignment_loss‚0.3726630356386035 2loss/FBt3/assignment_loss‚0.003117367185691617 1loss/SRqqt3/detection_loss‚0.9584778357725755 -loss/FBt1/detection_loss‚2.61909875699452 /loss/FBt2/detection_loss‚2.4126830661998078 /loss/FBt4/detection_loss‚1.6218881612193996 0loss/FRt4/assignment_loss‚2.5748564158167158 0loss/FBt2/assignment_loss‚0.5074509236017761 _runtime‚5765.480337366üÓ•Kì‚(:… †ÀëÔ­Ö§– gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.324' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.8648 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î¬Ótpí‚(:å †ÀëÔ·Ž‡— disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17' disk.nvme0n1.out‚121256.41943359375 disk.nvme0c0n1.in‚ 22029.6875$ disk.dm-0.out‚121256.41943359375 disk.sda1.in‚0$ memory_percent‚94.66214179992676' proc.memory.availableMB‚ 874.5546875 cpu‚25.29287216185836 network.sent‚ 1.1226464e+07 network.recv‚ 1.894624e+06$ disk./.usageGB‚1177.408405303955 disk.sda3.in‚0 disk.sda3.out‚ 184.90625) disk.nvme0c0n1.out‚121256.41943359375 disk.dm-1.in‚0 disk.dm-1.out‚ 184.90625) disk./.usagePercent‚79.01686599140207 disk.nvme0n1.in‚ 22029.6875 disk.dm-0.in‚ 22029.6875 disk.drbd1260.out‚ 0.8828125 disk.sda.out‚ 184.90625 disk.sda2.in‚0 disk.sda2.out‚0B\‚œî‚(:‘ •ÀëÔ߶ª˜ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.177' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.235400000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.à3‡ï‚(:ù •ÀëÔÛí¸œ$ disk./.usageGB‚1177.408405303955$ disk.dm-0.out‚121654.23583984375 disk.dm-1.in‚0$ memory_percent‚94.66063976287842" proc.memory.rssMB‚ 3037.5234375) disk.nvme0c0n1.out‚121654.23583984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 185.44140625 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.292413292428172 network.recv‚ 1.901334e+06) disk./.usagePercent‚79.01686599140207 disk.dm-0.in‚ 22169.8828125 disk.drbd1260.out‚ 0.8828125' disk.nvme0n1.out‚121654.23583984375 disk.sda3.in‚0) proc.memory.percent‚18.53957176208496{l@vý network.sent‚ 1.1335218e+07# disk.nvme0c0n1.in‚ 22169.8828125 disk.dm-1.out‚ 185.44140625 disk.sda.in‚0! disk.nvme0n1.in‚ 22169.8828125 disk.sda3.out‚ 185.44140625( proc.memory.availableMB‚ 874.80078125 proc.cpu.threads‚17 p@¡ð‚(:… ¤ÀëÔÉÆ— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚131.075' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.215 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê7M€ñ‚(:õ ¤ÀëÔöÛª™ disk.sda1.out‚0 network.sent‚ 1.133934e+07 network.recv‚ 1.904749e+06) disk./.usagePercent‚79.03364370116604 disk.sda3.out‚ 185.85546875# disk.dm-0.out‚122053.7685546875 disk.sda1.in‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 22169.8828125 disk.dm-0.in‚ 22169.8828125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.88671875 disk.sda.in‚0 disk.sda.out‚ 185.85546875) proc.memory.percent‚18.53957176208496 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 185.85546875 disk.sda2.out‚0! disk.nvme0n1.in‚ 22169.8828125# memory_percent‚94.6608304977417" proc.memory.rssMB‚ 3037.5234375 cpu‚25.291946825981526$ disk./.usageGB‚1177.658405303955( disk.nvme0c0n1.out‚122053.7685546875 disk.sda2.in‚0& disk.nvme0n1.out‚122053.7685546875( proc.memory.availableMB‚ 874.76953125fRkž‘ò‚(:† ³ÀëÔÀþÁ• gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë@ŠO‚ó‚(:÷ ³ÀëÔÏ匙 network.recv‚ 1.909435e+06 disk.drbd1260.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 22169.8828125& disk.nvme0n1.out‚122144.5224609375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 network.sent‚ 1.1345483e+07) disk./.usagePercent‚79.03206874281541$ disk./.usageGB‚1177.634937286377 disk.sda2.out‚0( disk.nvme0c0n1.out‚122144.5224609375 disk.dm-0.in‚ 22169.8828125# disk.dm-0.out‚122144.5224609375 disk.dm-1.out‚ 186.19921875 disk.sda.out‚ 186.19921875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 22169.8828125 disk.sda1.in‚0$ memory_percent‚94.66087818145752 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.88671875 disk.sda3.out‚ 186.19921875( proc.memory.availableMB‚ 874.76171875" proc.memory.rssMB‚ 3037.5234375) proc.memory.percent‚18.53957176208496 cpu‚25.291585332041414ÛúÞ”2ô- _wandbruntime‚5826  _runtime‚5826‡0–ÆõÀ _step‚381 & lr-AdamW/pg1‚0.0006262819276041897 & lr-AdamW/pg2‚0.0006262819276041897 trainer/global_step‚4699 $ _timestamp‚1.788534842075465e+09 _runtime‚5826.229087603PkD±ËöÅ _step‚381 & lr-AdamW/pg1‚0.0006262819276041897 & lr-AdamW/pg2‚0.0006262819276041897 trainer/global_step‚4699 $ _timestamp‚1.788534842075465e+09 _runtime‚5826.229087603ý¯GÞ2÷- _wandbruntime‚5826  _runtime‚5826$c‘­ß'øÙO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127471071 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190344520099840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚382 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9321048250368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199665586353152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9321048250368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216256914 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885348420766647e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88812189 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88812187 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88812187 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181237059654448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190344520099840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9320814581760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚ 5826.23035914 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211150475764224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199665334681600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52986016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52986018 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9480037379072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127444727 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127471071 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33747730 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190531692455772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127444727 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190344538102784 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199665334681600 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9294632801324ãô˜ä'ùÞO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199665334681600 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9294632801324 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127471071 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190344520099840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚382 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9321048250368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199665586353152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9321048250368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216256914 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885348420766647e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88812189 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88812187 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88812187 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181237059654448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190344520099840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9320814581760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚ 5826.23035914 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211150475764224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199665334681600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52986016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52986018 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9480037379072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127444727 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127471071 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33747730 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190531692455772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127444727 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190344538102784 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444þýrk²2ú- _wandbruntime‚5827  _runtime‚5827†ç|Ú›"ûäM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9323012289536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9322778607616 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190570935281738 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127497789 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88830462 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86751227 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127471419 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199706836429312 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216301881 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9323012289536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9322778607616 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216328253 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127471419 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9296362309226 _step‚383 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199706584744448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52996820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.fre¤'|‹Oed‚201711814087168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216301881 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52996818 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86751896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181274572972512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9482321484288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0bÈaï&üéM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211194159511552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190383806136832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885348433709857e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88830464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190383806136832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216328253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33754409 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216328253 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127471419 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9296362309226 _step‚383 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199706584744448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52996820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201711814087168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216301881 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52996818 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86751896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181274572972512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9482321484288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9323012289536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9322778607616 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190570935281738 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127497789 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88830462 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86751227 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127471419 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199706836429312 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216301881 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9323012289536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9322778607616 FËljÁ % _timestamp‚1.7885349024753983e+09 _runtime‚5886.629124278 _step‚385 & lr-AdamW/pg1‚0.0006212752627072054 & lr-AdamW/pg2‚0.0006212752627072054 trainer/global_step‚4749™<ÊiÌŠÆ & lr-AdamW/pg2‚0.0006212752627072054 trainer/global_step‚4749 % _timestamp‚1.7885349024753983e+09 _runtime‚5886.629124278 _step‚385 & lr-AdamW/pg1‚0.0006212752627072054’CæT2‹- _wandbruntime‚5886  _runtime‚5886—à'ŒÚO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚192505902347036 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201728103909376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89725937 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚192308839952384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89725939 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34095371 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚203754352559616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885349024769268e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53526118 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218508616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9419263956992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9579925539840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201727852237824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9419263956992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201727852237824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201728103909376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9580214062080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213334254159360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183113560486288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9392341860748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89725939 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89725937 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚203754328619520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚4749 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128782679 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128782679 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚213334566621696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9392570841832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9419030288384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53526116 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183113577525648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87621489 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192308821949440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9419030288384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34094694 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87620810 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚192506148367480Ñ;¶E ßO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183113577525648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87621489 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192308821949440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9419030288384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34094694 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87620810 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_by¾{…E tes.all.allocated‚192506148367480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚192505902347036 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201728103909376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89725937 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚192308839952384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89725939 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34095371 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚203754352559616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885349024769268e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53526118 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218508616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9419263956992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9579925539840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201727852237824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9419263956992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201727852237824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201728103909376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9580214062080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213334254159360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183113560486288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9392341860748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89725939 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89725937 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚203754328619520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚4749 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128782679 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128782679 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚213334566621696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9392570841832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9419030288384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53526116 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9420994314240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87638689 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89744214 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128809371 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53536918 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203796012589568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚4749 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203796036529664 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚192545391193446 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213377930740224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9581918150656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218579955 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218553583 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201769102300672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218553583 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192348125989376 _runtime‚5887.863135861 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34102443 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885349037083297e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218579955 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89744214 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192348107986432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192348107986432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89744212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192348125989376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89744212 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201769353985536 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53536920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87639363óÓb„ï&éM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9420994314240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87638689 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89744214 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128809371 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53536918 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203796012589568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚4749 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203796036529664 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚192545391193446 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213377930740224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9581918150656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218579955 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218553583 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201769102300672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218553583 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192348125989376 _runtime‚5887.863135861 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34102443 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885349037083297e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218579955 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89744214 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192348107986432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192348107986432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89744212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192348125989376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89744212 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201769353985536 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53536920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87639363 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183151090843712 _step‚387 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128835741 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128835741 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34101771 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚192545637214098 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183151107883072 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920ƒF¡º2‘- _wandbruntime‚5887  _runtime‚5887Œº|`’Ú .loss/FRt1/detection_loss‚2.165065812770539 /loss/FBt1/detection_loss‚3.2266062931580977 &loss/total_loss‚44.175506560018164 epoch‚3 _step‚388 0loss/FRt1/assignment_loss‚3.7865115725077114 1loss/FBt4/assignment_loss‚0.12589058279996265 /loss/FRt3/detection_loss‚1.6491344981427705 % _timestamp‚1.7885349037119443e+09 2loss/SRqqt3/assignment_loss‚0.9344525773247535 2loss/SRqqt4/assignment_loss‚1.0368626570410056 /loss/FBt3/detection_loss‚0.7904651074147806 2loss/FBt3/assignment_loss‚5.81508064032078e-08 /loss/FRt2/detection_loss‚2.9327560365200043 0loss/FRt2/assignment_loss‚3.6151241064071655 /loss/FRt4/assignment_loss‚3.367615549652665 2loss/FBt2/assignment_loss‚0.008282782389414933 0loss/SRqqt2/detection_loss‚3.030806490778923 trainer/global_step‚4749 _runtime‚5887.865415769 .loss/FRt4/detection_loss‚3.159346103668213 0loss/SRqqt1/detection_loss‚2.035871074387902 1loss/SRqqt3/detection_loss‚0.8740570766173981 2loss/SRqqt2/assignment_loss‚1.8822136413305999 /loss/FBt2/detection_loss‚2.4265061635524035 0loss/FRt3/assignment_loss‚1.3200146632535117 2loss/SRqqt1/assignment_loss‚1.3461550711409043 1loss/FBt1/assignment_loss‚0.19640795104367664 0loss/SRqqt4/detection_loss‚2.793370476235514 /loss/FBt4/detection_loss‚1.4719902137294412qqö«e“ß 0loss/SRqqt1/detection_loss‚2.035871074387902 1loss/SRqqt3/detection_loss‚0.8740570766173981 2loss/SRqqt2/assignment_loss‚1.8822136413305999 /loss/FBt2/detection_loss‚2.4265061635524035 0loss/FRt3/assignment_loss‚1.3200146632535117 2loss/SRqqt1/assignment_loss‚1.3461550711409043 1loss/FBt1/assignment_loss‚0.19640795104367664 0loss/SRqqt4/detection_loss‚2.793370476235514 /loss/FBt4/detection_loss‚1.4719902137294412 .loss/FRt1/detection_loss‚2.165065812770539 /loss/FBt1/detection_loss‚3.2266062931580977 &loss/total_loss‚44.175506560018164 epoch‚3 _step‚388 0loss/FRt1/assignment_loss‚3.7865115725077114 1loss/FBt4/assignment_loss‚0.12589058279996265 /loss/FRt3/detection_loss‚1.6491344981427705 % _timestamp‚1.7885349037119443e+09 2loss/SRqqt3/assignment_loss‚0.9344525773247535 2loss/SRqqt4/assignment_loss‚1.0368626570410056 /loss/FBt3/detection_loss‚0.7904651074147806 2loss/FBt3/assignment_loss‚5.81508064032078e-08 /loss/FRt2/detection_loss‚2.9327560365200043 0loss/FRt2/assignment_loss‚3.6151241064071655 /loss/FRt4/assignment_loss‚3.367615549652665 2loss/FBt2/assignment_loss‚0.008282782389414933 0loss/SRqqt2/detection_loss‚3.030806490778923 trainer/global_step‚4749 _runtime‚5887.865415769 .loss/FRt4/detection_loss‚3.159346103668213„¡ÍBy‘”‚(:† þÀëÔ‹É™ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ²EË|•‚(:ñ þÀëÔÚŠ†œ disk.sda1.out‚0$ memory_percent‚94.66266632080078% proc.memory.availableMB‚ 874.46875" proc.memory.rssMB‚ 3037.5234375 proc.cpu.threads‚17 network.sent‚ 1.1475352e+07 disk.sda.out‚ 189.02734375 disk.dm-0.in‚22325.35546875) proc.memory.percent‚18.53957176208496 cpu‚25.289431693057452 network.recv‚ 1.930694e+06$ disk./.usageGB‚1178.158405303955 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 189.02734375' disk.nvme0c0n1.out‚123686.228515625 disk.dm-1.in‚0' disk./.usagePercent‚79.067199120694 disk.drbd1260.out‚ 0.8984375" disk.nvme0n1.in‚22325.35546875 disk.sda3.in‚0$ disk.nvme0c0n1.in‚22325.35546875" disk.dm-0.out‚123686.228515625 disk.dm-1.out‚ 189.02734375 disk.sda1.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚123686.228515625ËK{–‚(:ð ÁëÔ ˜š proc.cpu.threads‚17 network.sent‚ 1.1584347e+07 network.recv‚ 1.937565e+06 disk.dm-0.in‚ 22360.6015625" disk.dm-0.out‚124147.251953125 disk.drbd1260.out‚ 0.8984375 disk.sda.in‚0 disk.sda.out‚ 189.55078125$ disk./.usageGB‚1178.158405303955 disk.dm-1.out‚ 189.55078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚94.66125965118408" proc.memory.rssMB‚ 3037.5234375' disk./.usagePercent‚79.067199120694# disk.nvme0c0n1.in‚ 22360.6015625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚124147.251953125( proc.memory.availableMB‚ 874.69921875) proc.memory.percent‚18.53957176208496' disk.nvme0c0n1.out‚124147.251953125! disk.nvme0n1.in‚ 22360.6015625 disk.sda3.out‚ 189.55078125 cpu‚25.28892795841405§m½‰—‚(:† ÁëÔµ¨¡§ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.469' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6938 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErü̦©rors‚0Òï+G˜‚(:… œÁëÔÕÝ£™ gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.657' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3314 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ymì/|™‚(:ñ œÁëÔ°‹œ disk.sda3.out‚ 190.0234375$ memory_percent‚94.66145038604736 proc.cpu.threads‚17 network.sent‚ 1.1589368e+07$ disk.nvme0c0n1.in‚22529.53515625! disk.dm-0.out‚124469.06640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.90234375( proc.memory.availableMB‚ 874.66796875" proc.memory.rssMB‚ 3037.5234375 network.recv‚ 1.940795e+06$ disk./.usageGB‚1177.767433166504& disk.nvme0c0n1.out‚124469.06640625 disk.dm-1.in‚0" disk.nvme0n1.in‚22529.53515625 disk.sda3.in‚0) proc.memory.percent‚18.53957176208496 cpu‚25.288551940149702 disk.dm-0.in‚22529.53515625 disk.dm-1.out‚ 190.0234375 disk.sda.in‚0 disk.sda2.in‚0) disk./.usagePercent‚79.04096065250216 disk.drbd1260.in‚0 disk.sda.out‚ 190.0234375 disk.sda2.out‚0$ disk.nvme0n1.out‚124469.06640625£ÙÝB‘š‚(:† «ÁëÔ¥œŒ— gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5612 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…ýºz›‚(:ï «ÁëÔîÏÛ› disk.sda.in‚0# disk.nvme0n1.out‚124776.1953125 disk.sda3.out‚ 190.35546875$ memory_percent‚94.66452598571777' proc.memory.availableMB‚ 874.1640625" proc.memory.rssMB‚ 3037.5234375 cpu‚25.28809194781481 network.recv‚ 1.94532e+06% disk./.usageGB‚1177.7674560546875% disk.nvme0c0n1.out‚124776.1953125 disk.drbd1260.out‚ 0.90234375 disk.sda.out‚ 190.35546875 disk.sda2.in‚0" disk.nvme0n1.in‚22529.53515625 proc.cpu.threads‚17 network.sent‚ 1.1595532e+07) disk./.usagePercent‚79.04096218854738$ disk.nvme0c0n1.in‚22529.53515625 disk.dm-0.out‚124776.1953125 disk.dm-1.out‚ 190.35546875 disk.sda1.in‚0 disk.sda3.in‚0) proc.memory.percent‚18.53957176208496 disk.dm-0.in‚22529.53515625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0Bt'¥2œ- _wandbruntime‚5948  _runtime‚5948Á5ØFľ _runtime‚ 5948.98079296 _step‚389 % lr-AdamW/pg1‚0.000616224215263567 % lr-AdamW/pg2‚0.000616224215263567 trainer/global_step‚4799 % _timestamp‚1.7885349648270934e+09@Tv•ÉžÃ % lr-AdamW/pg2‚0.000616224215263567 trainer/global_step‚4799 % _timestamp‚1.7885349648270934e+09 _runtime‚ 5948.98079296 _step‚389 % lr-AdamW/pg1‚0.000616224215263567…6¾ï2Ÿ- _wandbruntime‚5948  _runtime‚5948õ°]ß' ÙO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215518289759744 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚4799 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9517479663616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34443743 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚194480358258744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203790369794048 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130120631 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88509248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90639687 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194273141801984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9679762637824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9517245995008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220786664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90639689 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130146975 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9490279901256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203790621465600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205838527121920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184990078357488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90639689 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194273141801984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130120631 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9517245995008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194273123799040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205838551062016 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚390 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚203790369794048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215518600124928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184990095396848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚5948.982924057 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203790621465600 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9680049062912 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130146975 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203790369794048 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130120631 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88509248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90639687 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194273141801984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9679762637824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9517245995008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220786664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90639689 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130146975 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9490279901256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203790621465600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205838527121920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184990078357488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90639689 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194273141801984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130120631 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9517245995008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194273123799040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205838551062016 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104†-y‚ö2¢- _wandbruntime‚5950  _runtime‚5950¨;ëý #£äM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220831657 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34450373 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185027608714912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185027625754272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90657964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34449696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54077018 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194312427838976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130147323 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215561982928384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220805285 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚5950.106202988 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90657962 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194519847105154 trainer/global_step‚4799 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220831657 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90657964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54077020 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90657962 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9519210020864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220805285 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9519443702784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205880235032064 _step‚391 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130173693 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203831871541760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203831871541760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9492238390242 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88526714 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9492467371534 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0©‡Zï&¤éM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194312427838976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220831657 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90657964 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54077020 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90657962 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9519210020864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220805285 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9519443702784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205880235032064 _step‚391 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130173693 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203831871541760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203831871541760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9492238390242 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88526714 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9492467371534 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220831657 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34450373 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185027608714912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185027625754272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90657964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34449696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54077018 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194312427838976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130147323 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215561982928384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220805285 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚5950.106202988 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90657962 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194519847105154 trainer/global_step‚4799 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚187631124414144 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚385 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚187631124414144 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚385 BQ)¼ËR cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚197230008882688 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚197229990879744 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚218676654108672 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 223666019 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚92084964 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55049992 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9586606715034 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 161829376 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 trainer/global_step‚4835 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚92084964 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 131581055 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 131560344 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚388 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9613864015872 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚218676468339200 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚206843679083520 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚197229990879744 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55049990 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34981869 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚187656701103136 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206843872898560 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚90031473 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚197243307818170 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚206843872898560 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚34981483 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 % _timestamp‚1.7885350779407785e+09 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9613688203776 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9777708474880 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9586434894562 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚197243118958338 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223666019 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9777546645504 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚187656684063776 _runtime‚6062.094856857.%p#½ÐR OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9613688203776 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚208898945633792 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚92084962 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚206843679083520 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9613864015872 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223645306 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚90031861 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚92084962 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚386 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 131581055 _step‚394 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚197230008882688 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 131560344 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223645306 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 185769472 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34981869 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚187656701103136 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206843872898560 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚90031473 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚197243307818170 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚206843872898560 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚34981483 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 % _timestamp‚1.7885350779407785e+09 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9613688203776 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9777708474880 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9586434894562 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚197243118958338 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223666019 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9777546645504 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚187656684063776 _runtime‚6062.094856857 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚197230008882688 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚197229990879744 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚218676654108672 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 223666019 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚92084964 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55049992 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9586606715034 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 161829376 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 trainer/global_step‚4835 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚92084964 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 131581055 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 131560344^1ÎËæ MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚388 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9613864015872 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚218676468339200 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚206843679083520 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚197229990879744 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55049990 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5354166030883789 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5203754305839539 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007426303345710039 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.177610844373703 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5383240580558777 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06514713168144226 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15789473056793213 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9804339408874512 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15146198868751526 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43203866481781006 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17546437680721283 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2043195366859436 +jet/accuracy_3_of_3‚0.02249266393482685 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3766235411167145 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5142632126808167 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011111333966255188 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.46860039234161377 +particle/specificity‚0.9465118050575256 particle/accuracy_1_of_7‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49468672275543213 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4924571216106415 jet/accuracy_1_of_7‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08256532996892929 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1711767315864563 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.023425988852977753 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.973958432674408 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09810430556535721 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44591760635375977 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9806900024414062 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4728759825229645 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,particle/sensitivity‚0.28161486983299255 particle/accuracy_1_of_6‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9786801934242249 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5883620977401733 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04047619178891182 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3787878751754761 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6825491786003113 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2385164201259613 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.010394265875220299 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3597627282142639 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 jet/accuracy_4_of_6‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17465782165527344 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9903138875961304 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5633885860443115 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16648322343826294 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09263637661933899 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.03205128386616707 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13352200388908386 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347517371177673 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1747506856918335 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.41161611676216125 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.253401517868042 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.36816903948783875 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24537038803100586 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.43421515822410583 *jet/accuracy_1_of_1‚0.4708940088748932 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6775793433189392 'particle/f_score‚0.3583340048789978 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 +jet/accuracy_3_of_4‚0.06035783141851425 particle/accuracy_3_of_4‚1 (particle/accuracy‚0.8426854610443115 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23207595944404602 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4737979769706726 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1822943389415741 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.990472674369812 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36963605880737305 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4632987082004547 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9773706793785095 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.898448646068573 particle/accuracy_3_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28036394715309143 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6202771067619324 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 _step‚395 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25489380955696106 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7196969389915466 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22993044555187225 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5103238224983215 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.44930553436279297 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3980655372142792 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4725988209247589 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 +jet/accuracy_3_of_5‚0.06486175209283829 particle/accuracy_1_of_1‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052632331848145 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18345367908477783 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24734975397586823 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 -jet/accuracy_4_of_5‚0.0023041474632918835 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444328546524 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5650674104690552 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05763663351535797 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 particle/accuracy_3_of_7‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.402402400970459 jet/accuracy_5_of_5‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.43333330750465393 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_6_of_6‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18153023719787598 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4468238949775696 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008546251803636551 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4191302955150604 particle/accuracy_2_of_2‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48826467990875244 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5068009495735168 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2008938193321228 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0264858640730381 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5556477308273315 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1813211441040039 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1773620843887329 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 _runtime‚ 6080.48470056 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.015804598107933998 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2451900690793991 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16379310190677643 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,jet/accuracy_4_of_4‚0.002015621168538928 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920893907546997 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4363210201263428 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.46408048272132874 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3819940686225891 jet/accuracy_5_of_7‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.397659569978714 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99814772605896 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4677705466747284 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2081080973148346 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4233763515949249 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2757912874221802 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.35599133372306824 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4701659679412842 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6413690447807312 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5233333110809326 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2379959523677826 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0784725621342659 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46929144859313965 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1465633064508438 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5673990249633789 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.43754705786705017 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02103174850344658 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9200000166893005 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983085989952087 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4467054307460785 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5403186678886414 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19333334267139435 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19111059606075287 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23677366971969604 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9967001676559448 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1711767315864563 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028385326266288757 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_5_of_6‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2635278105735779 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4677705466747284 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991588294506073 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3658643364906311 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1985723227262497 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.43981480598449707 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9479167461395264 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2291666716337204 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9891042113304138 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8540778160095215 *jet/accuracy_2_of_5‚0.4464861750602722 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5107980370521545 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9809941649436951 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444328546524 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24115322530269623 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5317140221595764 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4486923813819885 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016900187358260155 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.32407405972480774 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2588841915130615 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16145125031471252 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2379629760980606 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.46746712923049927 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14094649255275726 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.53125 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18375465273857117 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.990845799446106 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2993793189525604 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4989035129547119 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.984316349029541 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979310631752014 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19109180569648743 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21060605347156525 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_7‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9898825287818909 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3345937430858612 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4539124667644501 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.37374430894851685 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 jet/accuracy_4_of_7‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42529764771461487 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 particle/accuracy_5_of_6‚1 jet/accuracy_3_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38137173652648926 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0929529145359993 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0173319261521101 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02091258205473423 7 Purity/Ù”à’1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0025767881888896227 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6502413749694824 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.45679011940956116 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4640090763568878 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13992224633693695 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_2‚0.7429021596908569 particle/accuracy_2_of_5‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5694738030433655 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.35699233412742615 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1784304976463318 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01547618955373764 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 *jet/accuracy_2_of_6‚0.4679487347602844 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9444443583488464 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16333916783332825 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5463104844093323 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16408047080039978 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 particle/accuracy_4_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 *loss/val_total_loss‚42.662100551487384 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4699740409851074 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4476736783981323 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9816400408744812 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9879464507102966 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15835578739643097 particle/accuracy_5_of_5‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4977421164512634 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 +jet/accuracy_2_of_3‚0.22059769928455353 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670634865760803 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.216379314661026 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_3‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44272562861442566 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_5_of_7‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_4‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3222222328186035 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5695692300796509 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4722222089767456 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.352287620306015 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4731646776199341 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9397163391113281 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5393369793891907 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945276975631714 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4100199341773987 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13368985056877136 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18028898537158966 particle/accuracy_1_of_4‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18036200106143951 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8143717646598816 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872041344642639 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0013227513991296291 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 trainer/global_step‚4835 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9298245310783386þçÈ`a¶¦ 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9773706793785095 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.898448646068573 particle/accuracy_3_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28036394715309143 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6202771067619324 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 _step‚395 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25489380955696106 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7196969389915466 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22993044555187225 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5103238224983215 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.44930553436279297 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3980655372142792 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4725988209247589 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 +jet/accuracy_3_of_5‚0.06486175209283829 particle/accuracy_1_of_1‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052632331848145 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18345367908477783 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24734975397586823 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 -jet/accuracy_4_of_5‚0.0023041474632918835 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444328546524 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5650674104690552 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05763663351535797 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 particle/accuracy_3_of_7‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.402402400970459 jet/accuracy_5_of_5‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.43333330750465393 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_6_of_6‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18153023719787598 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4468238949775696 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008546251803636551 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4191302955150604 particle/accuracy_2_of_2‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48826467990875244 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5068009495735168 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2008938193321228 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0264858640730381 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5556477308273315 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1813211441040039 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1773620843887329 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 _runtime‚ 6080.48470056 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.015804598107933998 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2451900690793991 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16379310190677643 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,jet/accuracy_4_of_4‚0.002015621168538928 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920893907546997 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4363210201263428 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.46408048272132874 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3819940686225891 jet/accuracy_5_of_7‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.397659569978714 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99814772605896 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4677705466747284 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2081080973148346 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4233763515949249 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2757912874221802 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.35599133372306824 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4701659679412842 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6413690447807312 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5233333110809326 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2379959523677826 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0784725621342659 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46929144859313965 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1465633064508438 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5673990249633789 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.43754705786705017 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02103174850344658 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9200000166893005 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983085989952087 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4467054307460785 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5403186678886414 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19333334267139435 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19111059606075287 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23677366971969604 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9967001676559448 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1711767315864563 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028385326266288757 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_5_of_6‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2635278105735779 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4677705466747284 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991588294506073 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3658643364906311 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1985723227262497 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.43981480598449707 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9479167461395264 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2291666716337204 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9891042113304138 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8540778160095215 *jet/accuracy_2_of_5‚0.4464861750602722 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5107980370521545 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9809941649436951 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444328546524 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24115322530269623 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5317140221595764 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4486923813819885 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016900187358260155 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.32407405972480774 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2588841915130615 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16145125031471252 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2379629760980606 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.46746712923049927 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14094649255275726 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.53125 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18375465273857117 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.990845799446106 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2993793189525604 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4989035129547119 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.984316349029541 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979310631752014 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19109180569648743 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21060605347156525 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_7‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9898825287818909 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3345937430858612 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4539124667644501 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.37374430894851685 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 jet/accuracy_4_of_7‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42529764771461487 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 particle/accuracy_5_of_6‚1 jet/accuracy_3_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38137173652648926 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0929529145359993 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0173319261521101 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02091258205473423 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0025767881888896227 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6502413749694824 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.45679011940956116 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4640090763568878 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13992224633693695 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_2‚0.7429021596908569 particle/accuracy_2_of_5‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5694738030433655 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.35699233412742615 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1784304976463318 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01547618955373764 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 *jet/accuracy_2_of_6‚0.4679487347602844 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9444443583488464 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16333916783332825 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5463104844093323 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16408047080039978 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 particle/accuracy_4_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 *loss/val_total_loss‚42.662100551487384 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4699740409851074 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4476736783981323 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9816400408744812 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9879464507102966 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15835578739643097 particle/accuracy_5_of_5‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4977421164512634 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 +jet/accuracy_2_of_3‚0.22059769928455353 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670634865760803 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.216379314661026 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_3‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44272562861442566 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_5_of_7‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_4‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3222222328186035 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5695692300796509 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4722222089767456 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.352287620306015 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4731646776199341 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9397163391113281 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5393369793891907 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945276975631714 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&PuriòådÝ1ty/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4100199341773987 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13368985056877136 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18028898537158966 particle/accuracy_1_of_4‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18036200106143951 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8143717646598816 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872041344642639 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0013227513991296291 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 trainer/global_step‚4835 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9298245310783386 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 epoch‚3 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37455809116363525 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3370833098888397 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01886792480945587 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9509280920028687 particle/accuracy_3_of_3‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 % _timestamp‚1.7885350963278306e+09 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.589115560054779 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6325014233589172 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48164311051368713 particle/accuracy_1_of_5‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9798941612243652 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24405844509601593 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5354166030883789 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5203754305839539 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007426303345710039 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.177610844373703 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5383240580558777 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06514713168144226 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15789473056793213 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9804339408874512 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15146198868751526 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43203866481781006 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17546437680721283 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2043195366859436 +jet/accuracy_3_of_3‚0.02249266393482685 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3766235411167145 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5142632126808167 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011111333966255188 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.46860039234161377 +particle/specificity‚0.9465118050575256 particle/accuracy_1_of_7‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49468672275543213 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4924571216106415 jet/accuracy_1_of_7‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08256532996892929 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1711767315864563 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.023425988852977753 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.973958432674408 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09810430556535721 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44591760635375977 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9806900024414062 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4728759825229645 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,particle/sensitivity‚0.28161486983299255 particle/accuracy_1_of_6‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9786801934242249 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5883620977401733 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04047619178891182 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3787878751754761 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6825491786003113 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2385164201259613 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.010394265875220299 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3597627282142639 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 jet/accuracy_4_of_6‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17465782165527344 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9903138875961304 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5633885860443115 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16648322343826294 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09263637661933899 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.03205128386616707 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13352200388908386 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347517371177673 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1747506856918335 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.41161611676216125 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.253401517868042 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.36816903948783875 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24537038803100586 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.43421515822410583 *jet/accuracy_1_of_1‚0.4708940088748932 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6775793433189392 'particle/f_score‚0.3583340048789978 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 +jet/accuracy_3_of_4‚0.06035783141851425 particle/accuracy_3_of_4‚1 (particle/accuracy‚0.8426854610443115 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23207595944404602 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4737979769706726 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1822943389415741 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.990472674369812 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36963605880737305 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4632987082004547‹qt:ˆÃ‚(:„ ÁÂëÔ­ôÝ™ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.88' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.176 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qNÍ%Ä‚(:ö ÁÂëÔèúÉœ disk.sda3.in‚0 disk.sda3.out‚ 195.2265625$ disk.dm-0.out‚127968.06982421875 disk.sda.in‚0$ disk./.usageGB‚1177.768466949463! disk.nvme0n1.in‚ 22802.3046875 disk.dm-0.in‚ 22802.3046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚94.67103481292725 cpu‚25.34069313704677 network.sent‚ 1.1856668e+07 disk.dm-1.in‚0 disk.dm-1.out‚ 195.2265625 disk.drbd1260.out‚ 0.92578125 disk.sda2.in‚0" proc.memory.rssMB‚ 3039.5078125) proc.memory.percent‚18.55168342590332 network.recv‚ 1.992103e+06# disk.nvme0c0n1.in‚ 22802.3046875) disk.nvme0c0n1.out‚127968.06982421875 disk.sda.out‚ 195.2265625 disk.sda2.out‚0( proc.memory.availableMB‚ 873.09765625 proc.cpu.threads‚17) disk./.usagePercent‚79.04103003054395' disk.nvme0n1.out‚127968.06982421875W…¾Ö2Å- _wandbruntime‚6099  _runtime‚6099WõÇÆÁ & lr-AdamW/pg1‚0.0006111298379039911 & lr-AdamW/pg2‚0.0006111298379039911 trainer/global_step‚4849 % _timestamp‚1.7885351156180727e+09 _runtime‚6099.771653247 _step‚3967Èž-ÌÇÆ _runtime‚6099.771653247 _step‚396 & lr-AdamW/pg1‚0.0006111298379039911 & lr-AdamW/pg2‚0.0006111298379039911 trainer/global_step‚4849 % _timestamp‚1.7885351156180727e+09ŒÕqò¨2È- _wandbruntime‚6099  _runtime‚6099R|8<à'ÉÚO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9645565547008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90484561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55342282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9618213122382 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198131581128704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132048068 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92522505 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35142277 trainer/global_step‚4849 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188476854155504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9645565547008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9645331878400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207777146675712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209835540041216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚6099.772808064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132048068 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198131563125760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35141623 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219645498936832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198094838296802 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9809958895616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9809670373376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207777146675712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224570573 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92522503 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188476871194864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92522503 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198131563125760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92522505 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55342284 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207776895004160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132021724 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90483905 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224544227 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209835516101120 % _timestamp‚1.7885351156191175e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198095084317246 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219645186474496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198131581128704ÀE'ýÊßO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207777146675712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224570573 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92522503 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188476871194864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92522503 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198131563125760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92522505 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55342284 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207776895004160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132021724 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90483905 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224544227 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209835516101120 % _timestamp‚1.7885351156191175e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198095084317246 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219645186474496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198131581128704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9645565547008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90484561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55342282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9618213122382 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198131581128704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132048068 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92522505 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35142277 trainer/global_step‚4849 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188476854155504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9645565547008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9645331878400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207777146675712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209835540041216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceSta